or: howto build a middlesized virtualized lab.
I’m setting up a demo for a medium sized virtualisation setup.
Key points of my plan were:
- unified storage
- abstraction of storage layer – be able to switch the storage layer (i.e. migrate from gluster to ceph or local storage)
- heavy memory overcommit
- abstraction of hypervisors
- cloud-like design, manage with opennebula
- being able to run multiple instances of “the same network”
Run one VM per Host that will own all local disks. The VM has glusterFS installed and we build a large networked storage from them. I don’t really like Gluster, it scales badly in IOPS and when compared to Ceph it’s as elegant as a military bunker. But it’s available and not in early testing stages.
For I need to be able to use page sharing (that is compress used memory pages, fake out unused pages and de-dupe duplicate pages) and use a fast 4k-page capable sandforce2 SSD.
Abstraction of hypervisors:
being able to switch or replace the hypervisor in use according to needs. It would be OK to convert or reinstall VMs, but basically you just want to save them as OVF and have grub / xen kernels set up.
Xen hosts must be mofified to have working pypxeboot.
If this is all done nicely you can have the same machine as a xen VM or virtualbox image on your laptop or on a real server. I won’t be taking prisoners at this point 🙂
Open Nebula all the way, but it relies on libvirt a lot, a dependency I cannot just do away with.
I wanted to go with Oracle VM because it is the only system with a non-broken Xen, but they don’t deliver much of a libvirt config. Oracle VM Manager is not really something I want to introduce here because the 2.x version is still slow and too limited.
run multiple versions of the same network:
I wanna enable the dev’s here to concurrently test different things with the same networks & servers
This is something done i.e. in the “virtualized cloud” project of a czech university. Having a QinQ capable switch around and optionally using OpenVswitch (can’t do qinq though) or Vyatta will surely help to get this working.
Here we will get the real problems with libvirt I think.
I’ll try to name based on a “cloud instance” i.e. c00-munich-net-02, same for the VM names.
A rough draft of the setup:
Now enter libvirt:
- need to have a default xml? (yes you can delete default network, but the next new one will use virbr0 AGAIN)
- need to define an ipv4 subnet for a network (bridge!!!)
- format for storage xml is stupid (define allocation = 0 if you’re using a directory, etc.)
- non-tls mode doesn’t work reliably
- storage pool driver randomly “not supported over this connection” with SSH, when using oracle vm.
- misleading errors when ssh keys aren’t setup
- defaulting to a desktop config with dnsmasq and virbr0
- You can’t just use an iso in a filesystem on the host, you have to define a storage pool for your isos
- Can’t reassign a VM’s network connection at all. (Or is that just a flaw of virt-manager? I doubt)
Why this is getting me so upset? Because half of those limitations come from bad design. And so they’re forcing anyone who doesn’t need their desktop-py ideas to follow them and bend everything.
After a day or two I saw I will have to first build the easy, featureless standard setup, using KVM bring up the networks and GlusterFS storage.
So for the moment, ditch Oracle VM, ditch Xen, ditch Stub IO domains and go with an easier setup. I have to remember most people would consider a 4-node GlusterFS Xen cluster as highly complicated and that KVM is mostly going so widespread because it fits into the nobrainer-desktop setups that libvirt assumes. I’ll have to build the easy thing and then slowly integrate the complex parts.
Now enter Ubuntu:
Yes, I’m just getting started….
Ubuntu Server forces me to set up a non-priv user account.
This is so f**** great: I’m at a customer site, neither do I have an account there nor am I allowed to. Get out of my WAY with your defaults. Just because it looks like a bright idea in your school network or whereever, it will not be sensible for everybody. So don’t force everybody to fit into your world.
So, after letting me set up a user account i’m not allowed to have, they put that user into /etc/sudoers and ENABLE HIM TO RUN ANY SHIT.
Wow GRATS to that, no root password is set and you recommend people to use sudo over su?
So there’s an unprivileged account that has permissions to do anything he likes if he remembers his own password. Instead of root’s password like with s su – c. So this is LESS secure than using su. WTF.
Did you ever read one of those unix security books? They always go to lengths about old unused maintenance accounts.
Awesomeness behold – I just learned where they come from 🙂
Also nice, I made a typo during setup and had a wrong port number for the http proxy. This was hardwired into apt.conf (nice actually). But why on earth does apt.conf take precedence over exporting http_proxy (which apt also supports). Ah well, I remember: GNU is not unix. Let’s just not do it the way it’s done. Variables override config files for 20 years? Not here :))