libvirt is so bad, why is everyone using it?

or: howto build a middlesized virtualized lab.

I’m setting up a demo for a medium sized virtualisation setup.

Key points of my plan were:

  1. unified storage
  2. abstraction of storage layer – be able to switch the storage layer (i.e. migrate from gluster to ceph or local storage)
  3. heavy memory overcommit
  4. abstraction of hypervisors
  5. cloud-like design, manage with opennebula
  6. being able to run multiple instances of “the same network”

Unified Storage:

Run one VM per Host that will own all local disks. The VM has glusterFS installed and we build a large networked storage from them. I don’t really like Gluster, it scales badly in IOPS and when compared to Ceph it’s as elegant as a military bunker. But it’s available and not in early testing stages.

Memory overcommit:

For  I need to be able to use page sharing (that is compress used memory pages, fake out unused pages and de-dupe duplicate pages) and use a fast 4k-page capable sandforce2 SSD.

Abstraction of hypervisors:

being able to switch or replace the hypervisor in use according to needs. It would be OK to convert or reinstall VMs, but basically you just want to save them as OVF and have grub / xen kernels set up.

Xen hosts must be mofified to have working pypxeboot.

If this is all done nicely you can have the same machine as a xen VM or virtualbox image on your laptop or on a real server. I won’t be taking prisoners at this point 🙂

Cloud-like design:

Open Nebula all the way, but it relies on libvirt a lot, a dependency I cannot just do away with.

I wanted to go with Oracle VM because it is the only system with a non-broken Xen, but they don’t deliver much of a libvirt config.  Oracle VM Manager is not really something I want to introduce here because the 2.x version is still slow and too limited.

run multiple versions of the same network:

I wanna enable the dev’s here to concurrently test different things with the same networks & servers

This is something done i.e. in the “virtualized cloud” project of a czech university. Having a QinQ capable switch around and optionally using OpenVswitch (can’t do qinq though) or Vyatta will surely help to get this working.

Here we will get the real problems with libvirt I think.

I’ll try to name based on a “cloud instance” i.e. c00-munich-net-02, same for the VM names.

A rough draft of the setup:

fully abstracted lab with networked storage

Now enter libvirt:

  • need to have a default xml? (yes you can delete default network, but the next new one will use virbr0 AGAIN)
  • need to define an ipv4 subnet for a network (bridge!!!)
  • format for storage xml is stupid (define allocation = 0 if you’re using a directory, etc.)
  • non-tls mode doesn’t work reliably
  • storage pool driver randomly “not supported over  this connection” with SSH, when using oracle vm.
  • misleading errors when ssh keys aren’t setup
  • defaulting to a desktop config with dnsmasq and virbr0
  • You can’t just use an iso in a filesystem on the host, you have to define a storage pool for your isos
  • Can’t reassign a VM’s network connection at all. (Or is that just a flaw of virt-manager? I doubt)

Why this is getting me so upset? Because half of those limitations come from bad design. And so they’re forcing anyone who doesn’t need their desktop-py ideas to follow them and bend everything.

After a day or two I saw I will have to first build the easy, featureless standard setup, using KVM bring up the networks and GlusterFS storage.

So for the moment, ditch Oracle VM, ditch Xen, ditch Stub IO domains and go with an easier setup. I have to remember most people would consider a 4-node GlusterFS Xen cluster as highly complicated and that KVM is mostly going so widespread because it fits into the nobrainer-desktop setups that libvirt assumes. I’ll have to build the easy thing and then slowly integrate the complex parts.

Now enter Ubuntu:

Yes, I’m just getting started….

Ubuntu Server forces me to set up a non-priv user account.

This is so f**** great: I’m at a customer site, neither do I have an account there nor am I allowed to. Get out of my WAY with your defaults. Just because it looks like a bright idea in your school network or whereever, it will not be sensible for everybody. So don’t force everybody to fit into your world.

So, after letting me set up a user account i’m not allowed to have, they put that user into /etc/sudoers and ENABLE HIM TO RUN ANY SHIT.

Wow GRATS to that, no root password is set and you recommend people to use sudo over su?

So there’s an unprivileged account that has permissions to do anything he likes if he remembers his own password. Instead of root’s password like with   s su – c. So this is LESS secure than using su. WTF.

Did you ever read one of those unix security books? They always go to lengths about old unused maintenance accounts.

Awesomeness behold – I just learned where they come from 🙂

Also nice, I made a typo during setup and had a wrong port number for the http proxy. This was hardwired into apt.conf (nice actually). But why on earth does apt.conf take precedence over exporting http_proxy (which apt also supports). Ah well, I remember: GNU is not unix. Let’s just not do it the way it’s done. Variables override config files for 20 years? Not here :))


2 thoughts on “libvirt is so bad, why is everyone using it?

  1. How’s your IO with glusterfs and Xen? I’m contemplating a similar build but have heard bad things about older versions of both, have matters improved with Xen4 and Gluster 3.1.X?

    • Hi,

      short version:
      got pissed off and will retry when i have spare time, rain and mood.

      long version:
      oh yes, i had performance issues
      i could not solve the mountroot issue that might or might not be related to direct IO mode
      the direct IO fixes in gluster might work or might not
      the fuse patch for gluster is no longer maintained and might still be needed or not
      fuse scaling out with 1m stripe size etc. needs more nodes than I can prep right away
      had connection startup issues –
      dns (meant to run without to avoid dependency on it) might be related or not
      was asked to check if my network was too slow. (i checked it out but they had no idea of IB and so noone would let me know if there was anything out of the ordinary. The values were 5micros/avg and 200micros/max)
      32bit is generally unsupported in gluster.
      i found that even via local loopback i had very low (<180MB/s) performance reading from 2 ssds.

      this is only a portion of the issues.

      the right strategy for this is to have time, approx 1 week undisturbed.
      build it all on something current like debian, ignore the xen bit until the performance is maxed out. have 8 nodes for testing, ideally all w/ 1 SSD.
      scale up, then re-apply testing with centos5/x64.
      Then go back to Oracle VM 32bit.

      I wanna point out, that oracle vm is not involved with the issues.
      30% is that my setup is too small, 20% is not enough IB clue and 50% is glusterFS general vagueness, documentation quality and so on.

      I had allotted 2 man-months for building gluster to the quality that I need and this proves more true than I had feared.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s