No-copy extracting Xen VM tarballs to LVM


SUSE Studio delivers Xen VM images which is really nice. They contain a sparse image and a (mostly incomplete) VM config file. Since I’m updating them pretty often I needed a hack that saves on any unneeed copies and needs no scratch space, either.

Goal: save copy times and improve life quality instead of copying and waiting…

First, lets have a look at the contents and then let’s check out how to directly extract them…

(Oh. Great. Shitbuntu won’t let me paste here)

 

Well, great.

I’n my case the disk image is called:

SLES_11_SP3_JeOS_Rudder_client.x86_64-0.0.6.raw

It’s located in a folder named:

SLES_11_SP3_JeOS_Rudder_client-0.0.6/

 

So, what we can do is this:

First, set up some variables so we can shrink the command later on…

version=0.0.6
appliance=SLES_11_SP3_JeOS_Rudder_client
url=https://susestudio.com/...6_64-${version}.xen.tar.gz
appliance=SLES_11_SP3_JeOS_Rudder_client
folder=${appliance}-${version}
vmimage=${appliance}.x86_64-${version}.raw
lv=/dev/vgssdraid5/lvrudderc1

Then, tie it together to store our VM data.

wget -O- $url | tar -O -xzf - ${folder}/${vmimage} | dd of=$lv bs=1024k

Storing to a file at the same time:

wget -O- $url | tee /dev/shm/myfile.tar.gz | tar -O -xzf - ${folder}/${vmimage} |\
dd of=$lv bs=1024k

 

Wget will fetch the file, write it to STDOUT, tar will read STDIN, only extract the image file, and write the extracted data to STDOUT, which is then buffered and written by the dd.

 

If you’ll reuse the image for multiple VMs like me you can also write it to /dev/shm and, if RAM allows, also gunzip it. the gzip extraction is actually limiting the performance, and even tar itself seems to be a little slow. I only get around 150MB/s on this.

I do remember it needs to flatten out the sparse image while storing to LVM, but I’m not sure if / how that influences the performance.

 

(Of course none of this would be necessary if the OSS community hadn’t tried to ignore / block / destroy standards like OVF as much as they could. Instead OVF is complex, useless and unsupported. Here we are.)

Blackhat 2014 talks you should really really look at


This is my watchlist compiled from the 2014 agenda, many of those talks are important if you want to be prepared of future and current issues.

Very great to see there’s also a few talks that fall more into the “defense” category.

 

# Talks concerning incredibly big and relevant issues. I filed those under “the world is gonna end”.

The first two are worthy of that and hopefully wake up people in the respective design bodies:

  • CELLULAR EXPLOITATION ON A GLOBAL SCALE: THE RISE AND FALL OF THE CONTROL PROTOCOL
  • ABUSING MICROSOFT KERBEROS: SORRY YOU GUYS DON’T GET IT

Also annoying to horrible threats

  • EXTREME PRIVILEGE ESCALATION ON WINDOWS 8/UEFI SYSTEMS
  • A PRACTICAL ATTACK AGAINST VDI SOLUTIONS
  • BADUSB – ON ACCESSORIES THAT TURN EVIL
  • A SURVEY OF REMOTE AUTOMOTIVE ATTACK SURFACES

 Things that will actually help improve security practices and should be watched as food for thought

  • OPENSTACK CLOUD AT YAHOO SCALE: HOW TO AVOID DISASTER
  • CREATING A SPIDER GOAT: USING TRANSACTIONAL MEMORY SUPPORT FOR SECURITYo
  • BUILDING SAFE SYSTEMS AT SCALE – LESSONS FROM SIX MONTHS AT YAHOO
  • BABAR-IANS AT THE GATE: DATA PROTECTION AT MASSIVE SCALE
  • FROM ATTACKS TO ACTION – BUILDING A USABLE THREAT MODEL TO DRIVE DEFENSIVE CHOICES
  • THE STATE OF INCIDENT RESPONSE

What could end our world five years from now:

  • EVASION OF HIGH-END IPS DEVICES IN THE AGE OF IPV6

note, memorize, listen to recommendations

  • HOW TO LEAK A 100-MILLION-NODE SOCIAL GRAPH IN JUST ONE WEEK? – A REFLECTION ON OAUTH AND API DESIGN IN ONLINE SOCIAL NETWORKS
  • ICSCORSAIR: HOW I WILL PWN YOUR ERP THROUGH 4-20 MA CURRENT LOOP
  • MINIATURIZATION

scada / modbus / satellites

  • THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED INJECTIONS
  • SATCOM TERMINALS: HACKING BY AIR, SEA, AND LAND
  • SMART NEST THERMOSTAT: A SMART SPY IN YOUR HOME
  • SVG: EXPLOITING BROWSERS WITHOUT IMAGE PARSING BUGS
  • THE BEAST WINS AGAIN: WHY TLS KEEPS FAILING TO PROTECT HTTP

Don’t recall what those two were about

  • GRR: FIND ALL THE BADNESS, COLLECT ALL THE THINGS
  • LEVIATHAN: COMMAND AND CONTROL COMMUNICATIONS ON PLANET EARTH

Xen Powermanagement


Hi all,

this is a very hot week and the sun is coming down on my flat hard. Yet, I’m not outside having fun: Work has invaded this sunday.

I ran into a problem: I need to run some more loaded VMs but it’s going to be hotter than usual. I don’t wanna turn into a piece of barbeque. The only thing I could do is to turn my Xen host’s powersaving features to the max.

Of course I had to write a new article on power management in the more current Xen versions from that… :)

Find it here: Xen Power management – for current Xen.

When I saved it I found, I also have an older one (which i wasn’t aware of anymore) that covers the Xen 3.4 era.

Xen full powersaving mode – for Xen 3.x

 

 

 

Trivia:
Did you know those settings only take a mouse click in VMWare?

Check_MK support for Allnet 3481v2


A friend of mine has had this thermometer and asked me to look into monitoring and setup.

I don’t think I ever put as much work into monitoring such a tiny device. Last evening and almost night I stabbed at it some more and finally completed the setup and documentation. I literally went to bed at 5am because of this tiny sensor.

To save others from this (and to make sure I have a reliable documentation for it…), I’ve made a wiki article out of the pretty tricky setup. Along the way I even found it still runs an old openssl.

You can check it out here:

http://confluence.wartungsfenster.de/display/Adminspace/Monitoring+Allnet+3418v2

The bitbucket version isn’t yet committed, I hope I will do this in a moment… :p
One interesting hurdle was I couldn’t do a check_mk package (using mkp list / mkp pack) since I also needed to include things from local/lib and similar folders. When I visit the MK guys again I’ll nag about this.

 

 

They have really pretty meters in their UI by the way.

Would hope something like it makes it to the nagvis exchange some day.

edit note: I initially wrote it has an “affected OpenSSL”. It seems they had built it back in 2012 without heartbeat, which is a nice and caring thing to do.
It’s still goddamn outdated.

Friday special: screenrc for automatic IRSSI start


Just wanted to share a little snippet.

This is my SSH+Screen config for my IRC box:

  • If I connect from any private system, I’ll get my irc window.
  • If it rebooted or something, the screen command will automatically re-create an IRSSI session for me.
  • If I detach the screen, i’m automatically logged out.
~$ cat .ssh/authorized_keys
command="screen -d -RR -S irc -U" ssh-[ key removed] me@mypc

The authorized keys settings enforce running only _this_ command, and the screen options set a title for, force-detach, force-reattach and force-create a screen session by the name “irc”.

~$ cat .screenrc 
startup_message off
screen -t irssi 1 irssi

The screenrc does the next step by auto-running irssi in win1 with title accordingly set.
(And it turns off the moronic GPL notice)
Irssi in itself is configured to autoconnect to the right networks and channels, of course. (to be honest: Irssi config is something I don’t like to touch more than every 2-3 years.)

On the clients I also have an alias in /etc/hosts for it, so if I type “ssh irc”, I’ll be right back on irc. Every time and immediately.

 

This is the tiny little piece of perfect world I was able to create, so I thought I’d share it.

FreeBSD periodic mails vs. monitoring


I love FreeBSD! Taking over a non-small infrastructure of around 75 FreeBSD servers was something I wouldn’t have wanted to pass on.

The problem bit is that I do consulting only, not pure ops. But there wasn’t much of an ops team left…

Where they used to put around 10 man-days per week into the feeding and care of FreeBSD plus some actual development, I’m now trying to do something in 1 day. And I still want it to be a well-run, albeit slower, ship.

One of the biggest hurdles was the sheer volume of email.

Adding up Zabbix alerts (70% of which concerning _nothing)), the FreeBSD periodic mails, cron outputs, and similar reporting I would see weeks with 1500+ mails or in the higher 1000s if there was any actual issues. Each week. Just imagine what it looked like when I didn’t visit my customer for 3 weeks…

Many of those mails have no point at all once You’re running more than -base:

The most typical example would be bad SSH logins. All those servers run software to block attackers and even feed that info back to a central authority and log there. So, why in hell would I want to know about malicious SSH connects?

Would you like a mail that tells you no hardware device has failed, today?

  • And another one every day until 2032?
  • From all servers?

This makes no sense.

Same goes for the mails that tell me about neccessary system updates.

What I’ve done so far can be put in those 3 areas:

1. Periodics:

Turn off as much of the periodic mails as possible (i.e. anything that is possible to see by other means). I tried to be careful with it, but it didn’t work like this. My periodic.conf looks like this now:

freebsd periodic.conf
I found turning off certain things like the “security mail” also disables portaudit DB updates. But I just changed my portaudit call to include the download. Somehow I had assumed that *update* would be separate from *report*.

2. Fix issues:

Apply fixes for any bugs that are really that, bugs. At least if I figure out how to fix them. More often than not I’ll hit a wall in between the NIH config management and bad perl code.

3. Monitor harder, but also smarter:

Put in better monitoring, write custom plugins for anything I need (OpenSSH Keys, Sendmail queues, OS Updates) and set thresholds to either a baseline value for “normal” systems or to values derived from peak loads for “busy” systems.

Some of the checks are to be found at my bitbucket, and honestly, I’m still constantly working on them.

https://bitbucket.org/darkfader/nagios/src/cc233b93c106166a5494d7488c38880df0a5946b/check_mk/freebsd_updates/?at=default

The checked in version might change quite often, I.e. I now think it won’t hurt to have a stronger separation of reporting for OS and Ports issues. And, maybe a check that tells me if I still need a reboot for a system.

The most current area now is automating the updates.

I’m taming the VMWare platform and using some Pysphere code to create VM snapshots on the fly. So there’s an ansible playbook that pulls updates. It’ll then check if there is a mismatch between the version reported from uname -a and the “tag” file from freebsd-update. In that case, it’ll trigger a VM snapshot and install / reboot.

Another piece of monitoring does a grep -R -e  “^<<<<<” -e “>>>>>” /etc and as such alerts me of unmerged files.

I try to do with tiny little pieces and have everything a dual-use (agriculture and weapons, you know) technology that gives me status reporting and status improvement.

I started a howto about the specifics I did in monitoring, see
FreeBSD Monitoring at my adminspace wiki.

Ansible FreeBSD update fun…


Using Ansible to make my time at the laundry place more interesting…

 

me@admin ~/playbooks]$ ansible-playbook -i hosts freebsd-updates.yml

PLAY [patchnow:&managed:&redacted-domain:!cluster-pri] *************

GATHERING FACTS ****************************************************
ok: [portal.dmz.redacted-domain.de]
ok: [carbon.dmz.redacted-domain.de]
ok: [irma-dev.redacted-domain-management.de]
ok: [lead.redacted-domain-intern.de]
ok: [polonium.redacted-domain-management.de]
ok: [silver.redacted-domain-management.de]
ok: [irma2.redacted-domain-management.de]
ok: [inoxml-89.redacted-domain-management.de]

TASK: [Apply updates] **********************************************
changed: [inoxml-89.redacted-domain-management.de]
changed: [carbon.dmz.redacted-domain.de]
changed: [portal.dmz.redacted-domain.de]
changed: [irma-dev.redacted-domain-management.de]
changed: [lead.redacted-domain-intern.de]
changed: [polonium.redacted-domain-management.de]
changed: [silver.redacted-domain-management.de]
changed: [irma2.redacted-domain-management.de]
 finished on lead.redacted-domain-intern.de
 finished on portal.dmz.redacted-domain.de
 finished on silver.redacted-domain-management.de
 finished on inoxml-89.redacted-domain-management.de
 finished on carbon.dmz.redacted-domain.de
 finished on polonium.redacted-domain-management.de
 finished on irma-dev.redacted-domain-management.de
 finished on irma2.redacted-domain-management.de

TASK: [Reboot] ****************************************************
changed: [carbon.dmz.redacted-domain.de]
changed: [portal.dmz.redacted-domain.de]
changed: [inoxml-89.redacted-domain-management.de]
changed: [irma-dev.redacted-domain-management.de]
changed: [lead.redacted-domain-intern.de]
changed: [polonium.redacted-domain-management.de]
changed: [silver.redacted-domain-management.de]
changed: [irma2.redacted-domain-management.de]

TASK: [wait for ssh to come back up] *******************************
ok: [portal.dmz.redacted-domain.de]
ok: [irma-dev.redacted-domain-management.de]

I now use a “patchnow” group to have some decision maker because *surprise* I don’t want to snapshot and patch all systems at once.

Quite annoying that the most fundamential admin decisions are always really tricky to put in automation systems (written by devs). Also, I’ll need to kick my own ass since the playbook didn’t trigger the snapshots anyway!

For the long term solution I think I’ll first define a proper policy based on factors like this:

  • How mature the installed OS version & patches are (less risk of patching)
  • How exposed the system is
  • The number of users affected by the downtime
  • The time needed for recovery

What factors do you look at?

Not enough desks to sufficiently bang your head on.


I’m not convinced. I have some LSI cards in several of my boxes and they
very consistently drop disks that are running SMART tests in the background.
I have yet to find a firmware or driver version that corrects this.

 

This is the kind of people I wish I never had to deal with.

(If not obvious: If your disks drop out while running SMART tests, look at the disks. There are millions of disks that handle this without issue. If yours drop out, they’re having a problem. Even if you think it doesn’t matter or even if it’s only showing with the controller. It doesn’t matter. Stuff it up.

I’m utterly sick of dealing with “admins” like this.)

Part four: Storage migration, disaster recovery and friends


This article also was published a little too early….. :)

 

A colleague (actually, my team lead) and I set out to build a new, FreeBSD based storage domU.

 

The steps we did:

Updated Raid Firmware

re-flashing my M5015 Raid controller to more current, non-IBM firmware. We primarly hoped this would enable the SSDs write cache. Didn’t work. It was a little easier than expected since I had already done parts of the procedure.

Your most important command for this is “Adpallinfo”

 

Created Raid Luns

We then created a large bunch of Raid10 luns over 4 of the SSDs.

  • 32GB for the storage domU OS
  • 512MB for testing a controller-ram buffered SLOG
  • 16GB ZIL
  • 16GB L2ARC
  • 600odd GB “rest”

Configure PCI passthrough in Xen

There was a few hickups, the kernel command line just wouldn’t activate, nor did using modprobe.d and /etc/modules do the job on their own.

This is what we actually changed…

First, we obtained the right PCI ID using lspci (apk add pciutils)

daveh0003:~# lspci | grep -i lsi

01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 03)

in /etc/modules:

xen_pciback

in /etc/modprobe.d/blacklist added:

blacklist megaraid_sas

in /etc/modprobe.d/xen-pciback.conf

options xen-pciback hide=(0000:01:00.0)

in /etc/update-extlinux.conf

default_kernel_opts=”modprobe.blacklist=megaraid_sas quiet” – we had also tried

#default_kernel_opts=”xen-pciback.hide='(01:00.0)’ quiet”

(btw, not escaping the paraentesis can cause busybox/openrc init to crash!!)

and, last, but not least I gave up annoyedly and put some stuff in /etc/rc.local

echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind

echo 0000:01:00.0 > /sys/bus/pci/drivers/pciback/new_slot

echo 0000:01:00.0 > /sys/bus/pci/drivers/pciback/bind

(and even this isn’t working without me manually calling it. It will take many more hours to get this to a state where it just works. If you ever wonder where the price of VMWare is justified… every-fucking-where)

FreeBSD storage domU

The storage domU is a pretty default install of FreeBSD10 to a 32GB LUN on the raid.

During install DHCP did not work ($colleague had also run into this issue) and so we just used a static IP… While the VM is called “freesd3″ I also added a CNAME called “stor” for easier access.

The zpools are:

  • zroot (the VM itself)
  • zdata (SSD-only)
  • zdata2 (Disk fronted by SSD SLOG and L2ARC)

I turned on ZFS compression on most of those using the dataset names, i.e.:

set compression=lz4 zroot/var

VMs can later access this using iSCSI or as a Xen block device (we’ll get to that later!)

Now, for the actual problem. During installation, the device IDs had shifted. On FreeBSD this is highly uncommon to see and you *really* consider that a linux-only issue. Well, not true.

Install

We selected “mfid0″, which should have been the 32GB OS Lun…

This is what MegaCli shows:

<<<megaraid_ldisks>>>
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Size                : 32.0 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 1 (Target Id: 1)
Size                : 3.182 TB
Sector Size         : 512
State               : Optimal
Strip Size          : 64 KB
Number Of Drives per span:2
Virtual Drive: 2 (Target Id: 2)
Size                : 512.0 MB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 3 (Target Id: 3)
Size                : 16.0 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 4 (Target Id: 4)
Size                : 64.0 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 5 (Target Id: 5)
Size                : 630.695 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2

Note that the logical drive ID and Target:Lun match up just fine!

 

 

The OS side:

Please compare to what FreeBSD’s mfi driver assigns…

mfid0: 32768MB (67108864 sectors) RAID volume (no label) is optimal
mfid1: 512MB (1048576 sectors) RAID volume (no label) is optimal
mfid2: 16384MB (33554432 sectors) RAID volume (no label) is optimal
mfid3: 65536MB (134217728 sectors) RAID volume (no label) is optimal
mfid4: 645832MB (1322663936 sectors) RAID volume (no label) is optimal
mfid5: 3337472MB (6835142656 sectors) RAID volume 'raid10data' is optimal

At install time it was cute enough to *drums* assign the 3.X T lun as mfid0. So we installed FreeBSD 10 on the LUN that stores my VMs.

That, of course, killed the LVM headers and a few gigabytes of data.

 

My next post will skip over reinstalling to the right lun (identified from live cd system) and instead describe how I went about getting the data back.

 

Heartbleed oder warum das Internet nicht mit “Windows” gemacht wird…


Gerade mit einem Chatbekannten ueber die Kundeninformation seiner Firma zu Heartbleed gesprochen…
20:17 <darkfader> hah und mit dem ssl
20:17 <darkfader> das lustige ist
20:18 <darkfader> weisst, das ist jetzt 7 tage her
20:18 <darkfader> vor 6 tagen waren wir unixer einigermassen fertig mit patchen
20:18 <darkfader> und audit vorher
20:18 <darkfader> und so
20:18 <darkfader> und die windows welt macht gerade zusammenfassungen der
betroffenen produkte

 

Erinnert mich sehr an eine Diskussion letztens ueber den Patchbedarf einer Unix-Umgebung

“Wenn das so stabil ist, warum muss man das dann so oft updaten?”

Von meiner Erklaerung hier mal ein kleines

Schaubild: