Jason Ruiz

Virtualization, Technology, and Stuff!

KVM: Nested Virtualization Support

This hit me like a sack of bricks, I looked for this before and never found any way of doing it, but I just happened to have found some articles from early last year that showed it was supported. As I’m writing this I’m getting a fully set up UEC cloud going, the proof is in the setup for this as it’ll flat out warn you 2-3 times that you don’t have KVM support, and it hasn’t yet. There are very few steps:

You’ll want to stop libvirtd in some instances, depending on what route you chose. Here we will remove and reinsert the module with nested support.

jason@Desktop:~$ sudo /etc/init.d/libvirt-bin stop
jason@Desktop:~$ sudo modprobe -r kvm_amd
jason@Desktop:~$ sudo modprobe kvm_amd nested=1

Next we will want to either add a new script to apparmor and the bin directory or change the current symlinked /usr/bin/kvm file. Let’s start off with the safe way, by creating separate script. This one I’ve named kvm-nested.

#!/bin/bash
exec /usr/bin/kvm -enable-nesting "$@"

Once that is done, you’ll need to edit the /etc/apparmor.d/abstractions/libvirt-qemu file and add the line below line into it in the section for “the various binaries”.

/usr/bin/kvm-nested rmix,

The one draw back to this method is you will need to manually edit each VM’s xml file to point to that new script /usr/bin/kvm-nested in this line.

 

<emulator>/usr/bin/kvm-nested</emulator>

The other way which I found more simple was deleting the current kvm symlink to qemu-system-x86_64 and pointing it to the script below.

jason@Desktop:~$ cat /usr/bin/kvm
#!/bin/bash
/usr/bin/qemu-system-x86_64 -enable-nesting "$@"
jason@Desktop:~$ ls -l /usr/bin/kvm
lrwxrwxrwx 1 root root 10 2011-01-21 19:13 /usr/bin/kvm -> /usr/bin/kvm-nested

That really should be it, now to keep testing to see if any issues arise.

 

**Update**

I wrote this article several years ago and had not touched KVM in just about as long. After starting to test OpenStack, I noticed that KVM now supports passing CPU flags via the -cpu argument. This means you can pass vmx to the guest by adding -cpu +vmx to the KVM argument list.

10 thoughts on “KVM: Nested Virtualization Support

  • bk says:

    Interesting article –
    what hypervisor have you nested under kvm and how is the performance?

  • Jason Ruiz says:

    Both KVM and Hyper-V work, ESX(i) is still a no. Performance isn’t that bad, I used Proxmox to deploy a few VM’s as a test and they ran decently. I really haven’t done any huge performance tests.

  • MO says:

    Jason, how did you get hyper-v to work ? I’ve tried to copy what you have here but hyper-v cannot start because the virtual machine bus fails to start with error code 39.

  • Jason Ruiz says:

    Well truthfully I hadn’t tested all of Hyper-V, I just wanted to make sure it was enabled as a proof of concept. With vSphere 5 coming within the next few months, running nested hypervisors should be more stable and have better compatibility.

  • MO says:

    OK. The situation seems to be that hyper-v will install but does not actually run, at least on my Ubuntu 10.04.2LTS box. I’ve tried W2K8 and W2K8R2 but both report the failure to start the virtual machine bus on the device manager after the hyper-v role is installed. Pity.

    Thanks for coming back to my question though as it helped me to know it is not fully operational.

  • […] Its possible, read more here – http://blog.coildomain.com/2011/01/24/kvm-nested-virtualization-support/ […]

    [WORDPRESS HASHCASH] The comment’s server IP (64.90.55.54) doesn’t match the comment’s URL host IP (64.90.54.59) and so is spam.

  • […] KVM on KVMをするための前提条件は、以下のサイトが参考になりました。 KVM: Nested Virtualization Support http://blog.coildomain.com/2011/01/24/kvm-nested-virtualization-support/ […]

  • Rodrigo Peluffo says:

    Hello Jason,

    I am building a lab with KVM over CentOs 6.5. So I have:

    CentOs with KVM > 2008R2 with Hyper-v > Error trying to install an Windows.

    I am trying to start a VM on Hyper-v but no sucess… Have you done it without issues? What is the best Linux to run KVM and allow nesting of hypervisors…?

    My focus is to be able to create a lab of Hyper-v and ESX, but till now I read that is not possible to run ESX nested on KVM.

    All my labs are over ESX nesting other hypervisors… but I would like implement it over a opensource solution…

    Thanks, see you…

  • Jason Ruiz says:

    I did run into the same problem, I had also tested this more recently and still came across it. Using the passthrough for VT-x with KVM is best used with KVM at this point.

  • Rodrigo Peluffo says:

    Hello Jason,

    I am looking for a solution to run ESX or Hyper-v as virtual Hypervisor…. I´ve tried proxmox but no success too… I guess AWS can run ESX or other hypervisor over but at this moment I just know that ESX and VMware Workstation are capable of run nested Hyper-v and ESX….

    Did you try to run XenServer with guests over KVM?

    see you…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>