• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

KVM vs XEN

T

tim_s

n00b
Joined
Nov 23, 2012
Messages
20
Hi Everyone,

I am trying to create a development server which was designed to imitate the conditions of our main production servers.

#
# SPECS
#

CPU: i7 3770
MEM: Corsair Vengeance 32GB
STORAGE: 2 x Samsung Pro in RAID 1

#
# WHAT I HAVE DONE SO FAR
#

*** Fresh install of Centos 6.4 minimal

Code: 
# yum install kvm

# yum install virt-manager libvirt libvirt-python python-virtinst

# modprobe kvm 

# modprobe kvm-intel

# /etc/init.d/libvirtd start 

# qemu-img create -f qcow2 -o preallocation=metadata /var/lib/libvirt/images/dev1.qcow2 60G

# virt-install --connect qemu:///system --name dev1 --ram 4096 --vcpus 4 --disk path=/var/lib/libvirt/images/dev1.qcow2,format=qcow2,bus=virtio,cache=none --network=bridge:br0,model=virtio --vnc --os-type=linux --os-variant=debiansqueeze --cdrom /var/lib/libvirt/images/debian-6.0.7-amd64-DVD-1.iso --accelerate --vnc --noautoconsole --keymap=en-us

#
# THE PROBLEM
#

- I notice frequently the CPU jumps up to 100 - 200%, the VM seems responsive until 200%.

- Shutting down the VM is challenging.

#
# QUESTION
#

I have noticed that XEN seems to have a huge following and I am looking to find something that is standardized. Is KVM not the solution when looking for a stable Virtual Machine?

I have used ESXi in the past and I have never had this poor of results.
 
KVM is a lot like Solaris LDOM's and in my experience tend to flake out if your network config isn't exactly spot on. This is where a mature platform like ESX really shines as they've already gotten past all the growing pains that plague KVM and LDOM's.

If you're going with KVM to avoid license fees and such associated with ESX or similar you may want to try Virtual Box to see if it'll meet your needs.
 
I tried xenxerver (and xcp) for a bit and found it fragile. What I mean is: very intolerant of something going wrong and leaving me with VMs that wouldn't boot, SRs that were unusable, etc...I really liked some of the features (particularly in xcp), but it just wasn't stable enough for me - too many sharp edges to cut myself on. I am using proxmox for kvm and it's fine.
 
Nothing I've experienced with my use of KVM, although my server guests tend to be 1-2 vcpus (and not terribly busy).

Is there a lot of IO going on in the guest? I could see a file backed guest starting to choke on IO, especially if the qcow image lives on an ext3 filesystem (ext3 serializes writes to a file, IIRC). Perhaps try LVM for your disk backing?

Another possibility is that the vcpus are being scheduled on 2 cores + the associated hyper threads (and the host is busy enough that the scheduler won't move those vcpu's around). CPU pinning is an option to work around this. It sounds like you're testing a single guest, though.

You could, of course, just try another hypervisor.
 
Last edited:
fantabulous said:
Nothing I've experienced with my use of KVM, although my server guests tend to be 1-2 vcpus (and not terribly busy).
Click to expand...

Oh for sure I am sure something is wrong on my side - it is not far fetched to throw this into the "miss" configuration pile.

fantabulous said:
Is there a lot of IO going on in the guest? I could see a file backed guest starting to choke on IO, especially if the qcow image lives on an ext3 filesystem (ext3 serializes writes to a file, IIRC). Perhaps try LVM for your disk backing?
Click to expand...

I am using LVM [Host/Guests], both in a Centos eviroment (Standard) and within Debian (Setup manually)

It is also important to note I am using RAID 1 as well on the host and this is on SSDs.

fantabulous said:
Another possibility is that the vcpus are being scheduled on 2 cores + the associated hyper threads (and the host is busy enough that the scheduler won't move those vcpu's around). CPU pinning is an option to work around this. It sounds like you're testing a single guest, though.
Click to expand...

HOST: Centos 6.4 (minimal)
GUEST 1: Centos 6.4 (minimal)
GUEST 2: Debian Squeeze (With Gnome 2)

In both cases the CPU sat at 100% or great was in installation and in updates. The rest of the time the CPU was quiet. My main issue is that the Debian Squeeze locked up and became unresponsive and I noticed the CPU was sitting @ 200%

fantabulous said:
You could, of course, just try another hypervisor
Click to expand...

I am thinking XEN.
 
We are transitioning from Xen to KVM in our staging and production machines, and I am running KVM in my colo servers and my server. The only time I've seen KVM wig out is when you over-allocate RAM and ksmtuned goes ballistic. Other than that, it's been a very stable platform for me.
 
Dark Shade said:
We are transitioning from Xen to KVM in our staging and production machines, and I am running KVM in my colo servers and my server. The only time I've seen KVM wig out is when you over-allocate RAM and ksmtuned goes ballistic. Other than that, it's been a very stable platform for me.
Click to expand...

This is the line I used to build the VM,

Code: 
# virt-install --connect qemu:///system --name dev1 --ram 4096 --vcpus 4 --disk path=/var/lib/libvirt/images/dev1.qcow2,format=qcow2,bus=virtio,cache=none --network=bridge:br0,model=virtio --vnc --os-type=linux --os-variant=debiansqueeze --cdrom /var/lib/libvirt/images/debian-6.0.7-amd64-DVD-1.iso --accelerate --vnc --noautoconsole --keymap=en-us

On board we have 32 GB of RAM, which is visible when querying proc. (Can't show this as I do not have remote access right now)
 
I'm not sure if it matters at all, but here's my personal script for starting KVM VMs:


Code: 
#!/bin/bash
VM_NAME=Testmachine
IPADDR=10.0.0.10

virt-install \
        --name ${VM_NAME} \
        --ram 2048 \
        --vcpus=2 \
        --disk path=/var/lib/libvirt/images/${VM_NAME}.img \
        -w bridge:br0 \
        -w bridge:br1 \
        --arch x86_64 \
        -l http://kickstart.local/CentOS/6/os/x86_64 --graphics none \
        --extra-args='console=tty0 console=ttyS0,115200n8 serial ks=http://kickstart.local/kickstart/${VM_NAME}
        ksdevice=eth0 ip=${IPADDR} gateway=10.0.0.1 netmask=255.255.255.0 dns=10.0.0.2' -v \

I watch the processes using htop and occasionally, virt-manager
 
Not receiving errors but it won't shut down, you mean when it's done kickstarting? Sounds like something in the postinstall is hanging.
 
Dark Shade said:
Not receiving errors but it won't shut down, you mean when it's done kickstarting? Sounds like something in the postinstall is hanging.
Click to expand...

Sorry Dark Shade I really omitted a lot of detail which lead to this confusion.

#
# My compliments
#

Your scripts looks great but since I have not referenced kickstart in any of my VM Installs I will need to read up on it.

#
# My problem
#

I have two issues, 1: The CPU sits at 100% - 200%, whenever the hard drive has a lot of activity.

First noticed during a Centos installation where the CPU rose up to 100%.

Second noticed when I ran a software update through the debian package manager. The CPU ran up to 200% and the VM of course locked up.

Third and I am not entirely sure what drives this behaviour, the VM does not respond to,

virsh shutdown dev1

where dev1 is the reference name of the instance.
 
If the CPU of the virtual machine is pegged at 100%+, virsh shutdown may not bring it down. It would be like pegging a physical machine and making it incredible unresponsive, then pressing the Power button -- it triggers the shutdown sequence on the machine to do a safe shutdown, but since all of the processing is being held up, it takes forever (if ever) to complete the shutdown process.

A 'virsh destroy vmname' will bring it down (akin to unplugging it). In KVM, you can kill -9 the process running the VM as well, though I have untested this to speak of its reliability.

Have you tried different settings, like using raw .img disks instead of qcow2, or disabling vnc?
 
Dark Shade said:
If the CPU of the virtual machine is pegged at 100%+, virsh shutdown may not bring it down. It would be like pegging a physical machine and making it incredible unresponsive, then pressing the Power button -- it triggers the shutdown sequence on the machine to do a safe shutdown, but since all of the processing is being held up, it takes forever (if ever) to complete the shutdown process.

A 'virsh destroy vmname' will bring it down (akin to unplugging it). In KVM, you can kill -9 the process running the VM as well, though I have untested this to speak of its reliability.
Click to expand...

This is sensible - It is what I figure excluding that this issue can happen at anytime regardless of CPU utilization. I.e. I can see the CPU sitting at 0.3% idling almost and and still have issues shutting down the machine. I maybe impatient, I am using virsh list to determine status - I do not have access to the machine but I see the following results.

# virsh list

dev1 is running

# virsh shutdown dev1

dev1 is shutting down ....

# virsh list

dev1 is running

# virsh list

dev1 is running

etc etc. - I never see it "NOT" running and this result is not consistent.

Dark Shade said:
Have you tried different settings, like using raw .img disks instead of qcow2, or disabling vnc?
Click to expand...

I have thought about RAW as well - The challenge with disabling VNC is that the frontend support machine is Windows and I am unable to run virt manager. So I am stuck with finding a method of completing the installation without it.

Here is some notes I have taken,

"--ram 4096"

I have 32 GB of Ram - perhaps I am not understanding the unit of measurement. To me I read it as MB.

"--vcpus 4"

Since the Host requires CPU space and the i7 only has 4 true cores (8 with hyperthreading) this could be causing a reaction.

"--disk path=/var/lib/libvirt/images/dev1.qcow2,format=qcow2,bus=virtio,cache=none"

Probably should of used RAW over qcow2.

"I am using a RAID 1 on the host"

This is not a duplication of the production servers

"I am using SSD on the host"

This is the first time I have done this.
 
The RAM is in MB, so you're right, 4096=4GB

You could install Windows using PXEboot images so they are entirely unattended, but doing manual installs will require VNC, and I've not had problems using the VNC method, just throwing ideas around to find possible culprits.

The guests don't care what kind of RAID level it is, or type of physical disk (spinning or SSD), as long as you give it an image, but the type of image may have an impact (qcow2/qed/raw/etc)

Have you tried using virt-manager to do an install, using the very basic default settings it offers? Just for a small host to use for testing?
 
Dark Shade said:
The RAM is in MB, so you're right, 4096=4GB

The guests don't care what kind of RAID level it is, or type of physical disk (spinning or SSD), as long as you give it an image, but the type of image may have an impact (qcow2/qed/raw/etc)
Click to expand...

Yeah I will try RAW next. just 200% on such a low class VM seems extreme but I will post my findings.

Dark Shade said:
Have you tried using virt-manager to do an install, using the very basic default settings it offers? Just for a small host to use for testing?
Click to expand...

My understanding is that virt-manager is GUI only. I have a headless machine and thus administration is happening on a Windows machine. I do not think virt-manager exists within the Windows environment.

I maybe wildly wrong and I would love to know more.

--

What I am going to do is run your script for creating VMs, modify it slightly so I can control RAM and CPU and add the "--vnc" switch to trigger this. You are already using the .img format so we are probably good there and I will post the results.

"-l http://kickstart.local/CentOS/6/os/x86_64"

This is the information I need to understand and then I should be good to go.
 
If you have your x11 stuff installed on the host machine, you can use XMing to open and display X windows on Windows. I normally PuTTY into the host machine with the X11 settings configured (localhost:0.0) and xming running on Windows. virt-manager command should bring it up, like so:

virtmanager.png
 
Oh I see, the kickstart - I understand now.

Actually IF you can - want to throw me an example of your kickstart?
 
Dark Shade said:
If you have your x11 stuff installed on the host machine, you can use XMing to open and display X windows on Windows. I normally PuTTY into the host machine with the X11 settings configured (localhost:0.0) and xming running on Windows. virt-manager command should bring it up, like so:

virtmanager.png
Click to expand...

Nice - I am using MobXterm but I could switch back to putty. I do not have x11 installed on my machine but this is not a major issue either. I will look into XMing.

This is very helpful.
 
Example of a kickstart I use:

Code: 
install
url --url http://kickstart.local/CentOS/6/os/x86_64
lang en_US.UTF-8
keyboard us
network --device eth0 --bootproto static --ip 10.0.0.100 --netmask 255.255.255.0 --gateway 10.0.0.1 --nameserver 10.0.0.2 --hostname data.local
rootpw --iscrypted SALTED_AND_ENCRYPTED_PW_HERE
firewall --enabled --port=22:tcp
authconfig --enableshadow --enablemd5
selinux --enforcing
timezone --utc America/New_York
bootloader --location=mbr --driveorder=vda --append="console=xvc0"
reboot

# Partitioning
zerombr yes
clearpart --all --initlabel --drives=vda
part /boot --fstype ext4 --size=100 --ondisk=vda
part pv.2 --size=1 --grow --ondisk=vda
volgroup data--pesize=32768 pv.2
logvol / --fstype ext4 --name=root --vgname=data--size=1024 --grow
logvol swap --fstype swap --name=swap --vgname=data--size=1024 --maxsize=1024

%packages
@base
@core
@network-server
@web-server
nmap
lftp
gcc
openssl-devel
xinetd
telnet
make
screen
any_other_package_from_yum

#postinstall -- any bash script you need to install things, like nagios nrpe
%post
chvt 3
cd /tmp
wget http://kickstart.local/kickstart/files/postinstall
bash postinstall > /root/postinstall.log
chvt 1
 
tim_s said:
Third and I am not entirely sure what drives this behaviour, the VM does not respond to,

virsh shutdown dev1

where dev1 is the reference name of the instance.
Click to expand...

Just wanted to mention that there's the possibility you're missing something like acpi scripts. I can't remember the package name, but I've had this issue with Debian based guests. I'm assuming the guest is totally unresponsive via SSH, though.

tim_s said:
I have thought about RAW as well - The challenge with disabling VNC is that the frontend support machine is Windows and I am unable to run virt manager. So I am stuck with finding a method of completing the installation without it..
Click to expand...

In regards to RAW and my previous comment about using LVM as the backing disk, I wasn't terribly clear about what I meant. My suggestion was to use something like:

"--disk path=/dev/my_vg/my_lv"

That might require some resizing of your host logical volumes to create enough free space. It's potentially terribly wasteful of disk space compared to sparse files as well. Something to think about if you still have issues with file based disk images, and you don't just try out another hypervisor.

I don't think having the VNC server enabled would cause the issues, you're seeing, but do want to mention that you can use (virtual) serial consoles with your KVM guests and run through text mode installers via SSH that way. It'd be something like this at the end of your virt-install command:

--extra-args="serial console=tty0 console=ttyS0,115200"

Too lazy to catch the last quote, but do want to mention just for safety that Debian uses preseed rather than kickstart. Same concept, different enough syntax.
 
Hi Everyone,

Some feedback:

So I came home and realized actually MobaXTerm can handle X11 forwarding out of the box. So actually it was just a case of installing X11, Loggin into SSH and typing "virt-manager"

I noticed the Guest "dev1" was shut off and so I enabled the guest and everything was smooth. I logged into Gmail to see it's response and immediately the host shot up to 103% and the guest locked up. The good news, virt-manager had zero issues stopping the VM.

CPU usage was low and oddly enough TOP and Host CPU usage disagree with the performance of the machine.

More odd: When clicking view, Disk I/O and Network I/O is disabled.

I will be creating a brand new guest in a moment and will report on the setup.
 
So I have installed a guest via virt-manager and the performance is much better. -- I will need to see what the difference is.
 
Yeah - just using virt-manager resolved the problem.

I have 4 VMs running right now installing and everything is running smoothly, including the fact I have re-created the Debian Squeeze machine.
 
Mtnduey said:
KVM is a lot like Solaris LDOM's and in my experience tend to flake out if your network config isn't exactly spot on. This is where a mature platform like ESX really shines as they've already gotten past all the growing pains that plague KVM and LDOM's.

If you're going with KVM to avoid license fees and such associated with ESX or similar you may want to try Virtual Box to see if it'll meet your needs.
Click to expand...

KVM's been solid for us (as have our Solaris LDoms for that matter). We've since migrated off KVM to RHEV as we're a Red Hat shop and want the full support. Our KVM hosts ran well and gave us no real problems before they were migrated.
 
I use kill -9 all the time to blow away kvm virtual machines and haven't had any issues so far.

If you want kvm, but prefer a gui and baremetal setup like esxi then try out Proxmox
 
+1 on proxmox. You can get free clustering and live migration although not (yet) storage migration.
 
Everything is looking really positive and stable.

I currently have 5 VMs running but I am noticing a lot of Memory allocation and hitting Swap.

I have the following,

# DB (Centos, Mysql)

VCPU: 4
RAM: 4 GB
HD: 1 Virtio Disks

# Debian (with GUI)

VCPU: 2
RAM: 4 GB
HD: 1 Virtio Disks

# SAN (Centos iSCSI)

VCPU: 2
RAM: 4 GB
HD: 7 Virtio Disks

# WEB (Cento Apache)

VCPU: 2
RAM: 2 GB
HD: 1 Virtio Disks

# DNS (Centos Bind)

VCPU: 1
RAM: 1 GB
HD: 1 Virtio Disk

With all these virtually idle has the the 32GB system hitting SWAP already, which seems odd.

I probably could reduce the SAN / Debian machine by 2 GB each. They will just have a drop in performance - most large transactions are backups at night.
 
Have you ever tried Xen? I can't say much about KVM as I never tried it since it lacked some features I needed.

I've found Xen to be quite easy to work with, though my primary use of it is running a Windows 7 guest with PCI and VGA passthrough. Performance of the Windows guest is top notch.

I have good experiences with LVM, including striped LVM disks, but I use RAW storage. That may make a difference.

Did you enable the VT-d (IOMMU) option in the motherboard BIOS? Not sure how KVM makes use of it, but it might help.

Good luck!
 
You must log in or register to reply here.
Back
Top