Time for 2016


Hi everyone.

 

I just thought some post is in place after having gone dark for quite a long time.

I’d been home sick for almost a month. First I’d snapped my back very badly and then I caught a strong flu. This completely ruined the month I had meant to spend on posting things here. Lying on your back is BORING and painkillers make you too numb do to anything useful.

Between that I’ve also done a few fun projects and been to OpenNebulaConf (Oct), the chaos communication camp (Dec) and the config management camp (Feb) and each time I came home with some nice ideas to throw around.

To be honest though, the highlight of the last months was watching Deadpool.

If you can handle some completely immature humor and the good old ultraviolent, go watch it.

 

For this year, there will be EuroBSDCon and OpenNebulaConf yet again.

One great thing about OpenNebula is the extremely friendly community. Comparing this to *any* other conf I’ve been to they are all pretty darn hostile and bro-ish. OpenNebula is such a nice community in  comparison and I really hope the others will start trying to match up with that at some point.

Blackhat 2014 talks you should really really look at


This is my watchlist compiled from the 2014 agenda, many of those talks are important if you want to be prepared of future and current issues.

Very great to see there’s also a few talks that fall more into the “defense” category.

 

# Talks concerning incredibly big and relevant issues. I filed those under “the world is gonna end”.

The first two are worthy of that and hopefully wake up people in the respective design bodies:

  • CELLULAR EXPLOITATION ON A GLOBAL SCALE: THE RISE AND FALL OF THE CONTROL PROTOCOL
  • ABUSING MICROSOFT KERBEROS: SORRY YOU GUYS DON’T GET IT

Also annoying to horrible threats

  • EXTREME PRIVILEGE ESCALATION ON WINDOWS 8/UEFI SYSTEMS
  • A PRACTICAL ATTACK AGAINST VDI SOLUTIONS
  • BADUSB – ON ACCESSORIES THAT TURN EVIL
  • A SURVEY OF REMOTE AUTOMOTIVE ATTACK SURFACES

 Things that will actually help improve security practices and should be watched as food for thought

  • OPENSTACK CLOUD AT YAHOO SCALE: HOW TO AVOID DISASTER
  • CREATING A SPIDER GOAT: USING TRANSACTIONAL MEMORY SUPPORT FOR SECURITYo
  • BUILDING SAFE SYSTEMS AT SCALE – LESSONS FROM SIX MONTHS AT YAHOO
  • BABAR-IANS AT THE GATE: DATA PROTECTION AT MASSIVE SCALE
  • FROM ATTACKS TO ACTION – BUILDING A USABLE THREAT MODEL TO DRIVE DEFENSIVE CHOICES
  • THE STATE OF INCIDENT RESPONSE

What could end our world five years from now:

  • EVASION OF HIGH-END IPS DEVICES IN THE AGE OF IPV6

note, memorize, listen to recommendations

  • HOW TO LEAK A 100-MILLION-NODE SOCIAL GRAPH IN JUST ONE WEEK? – A REFLECTION ON OAUTH AND API DESIGN IN ONLINE SOCIAL NETWORKS
  • ICSCORSAIR: HOW I WILL PWN YOUR ERP THROUGH 4-20 MA CURRENT LOOP
  • MINIATURIZATION

scada / modbus / satellites

  • THE NEW PAGE OF INJECTIONS BOOK: MEMCACHED INJECTIONS
  • SATCOM TERMINALS: HACKING BY AIR, SEA, AND LAND
  • SMART NEST THERMOSTAT: A SMART SPY IN YOUR HOME
  • SVG: EXPLOITING BROWSERS WITHOUT IMAGE PARSING BUGS
  • THE BEAST WINS AGAIN: WHY TLS KEEPS FAILING TO PROTECT HTTP

Don’t recall what those two were about

  • GRR: FIND ALL THE BADNESS, COLLECT ALL THE THINGS
  • LEVIATHAN: COMMAND AND CONTROL COMMUNICATIONS ON PLANET EARTH

Friday special: screenrc for automatic IRSSI start


Just wanted to share a little snippet.

This is my SSH+Screen config for my IRC box:

  • If I connect from any private system, I’ll get my irc window.
  • If it rebooted or something, the screen command will automatically re-create an IRSSI session for me.
  • If I detach the screen, i’m automatically logged out.
~$ cat .ssh/authorized_keys
command="screen -d -RR -S irc -U" ssh-[ key removed] me@mypc

The authorized keys settings enforce running only _this_ command, and the screen options set a title for, force-detach, force-reattach and force-create a screen session by the name “irc”.

~$ cat .screenrc 
startup_message off
screen -t irssi 1 irssi

The screenrc does the next step by auto-running irssi in win1 with title accordingly set.
(And it turns off the moronic GPL notice)
Irssi in itself is configured to autoconnect to the right networks and channels, of course. (to be honest: Irssi config is something I don’t like to touch more than every 2-3 years.)

On the clients I also have an alias in /etc/hosts for it, so if I type “ssh irc”, I’ll be right back on irc. Every time and immediately.

 

This is the tiny little piece of perfect world I was able to create, so I thought I’d share it.

Part four: Storage migration, disaster recovery and friends


This article also was published a little too early….. 🙂

 

A colleague (actually, my team lead) and I set out to build a new, FreeBSD based storage domU.

 

The steps we did:

Updated Raid Firmware

re-flashing my M5015 Raid controller to more current, non-IBM firmware. We primarly hoped this would enable the SSDs write cache. Didn’t work. It was a little easier than expected since I had already done parts of the procedure.

Your most important command for this is “Adpallinfo”

 

Created Raid Luns

We then created a large bunch of Raid10 luns over 4 of the SSDs.

  • 32GB for the storage domU OS
  • 512MB for testing a controller-ram buffered SLOG
  • 16GB ZIL
  • 16GB L2ARC
  • 600odd GB “rest”

Configure PCI passthrough in Xen

There was a few hickups, the kernel command line just wouldn’t activate, nor did using modprobe.d and /etc/modules do the job on their own.

This is what we actually changed…

First, we obtained the right PCI ID using lspci (apk add pciutils)

daveh0003:~# lspci | grep -i lsi

01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 03)

in /etc/modules:

xen_pciback

in /etc/modprobe.d/blacklist added:

blacklist megaraid_sas

in /etc/modprobe.d/xen-pciback.conf

options xen-pciback hide=(0000:01:00.0)

in /etc/update-extlinux.conf

default_kernel_opts=”modprobe.blacklist=megaraid_sas quiet” – we had also tried

#default_kernel_opts=”xen-pciback.hide='(01:00.0)’ quiet”

(btw, not escaping the paraentesis can cause busybox/openrc init to crash!!)

and, last, but not least I gave up annoyedly and put some stuff in /etc/rc.local

echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind

echo 0000:01:00.0 > /sys/bus/pci/drivers/pciback/new_slot

echo 0000:01:00.0 > /sys/bus/pci/drivers/pciback/bind

(and even this isn’t working without me manually calling it. It will take many more hours to get this to a state where it just works. If you ever wonder where the price of VMWare is justified… every-fucking-where)

FreeBSD storage domU

The storage domU is a pretty default install of FreeBSD10 to a 32GB LUN on the raid.

During install DHCP did not work ($colleague had also run into this issue) and so we just used a static IP… While the VM is called “freesd3” I also added a CNAME called “stor” for easier access.

The zpools are:

  • zroot (the VM itself)
  • zdata (SSD-only)
  • zdata2 (Disk fronted by SSD SLOG and L2ARC)

I turned on ZFS compression on most of those using the dataset names, i.e.:

set compression=lz4 zroot/var

VMs can later access this using iSCSI or as a Xen block device (we’ll get to that later!)

Now, for the actual problem. During installation, the device IDs had shifted. On FreeBSD this is highly uncommon to see and you *really* consider that a linux-only issue. Well, not true.

Install

We selected “mfid0”, which should have been the 32GB OS Lun…

This is what MegaCli shows:

<<<megaraid_ldisks>>>
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Size                : 32.0 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 1 (Target Id: 1)
Size                : 3.182 TB
Sector Size         : 512
State               : Optimal
Strip Size          : 64 KB
Number Of Drives per span:2
Virtual Drive: 2 (Target Id: 2)
Size                : 512.0 MB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 3 (Target Id: 3)
Size                : 16.0 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 4 (Target Id: 4)
Size                : 64.0 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Virtual Drive: 5 (Target Id: 5)
Size                : 630.695 GB
Sector Size         : 512
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2

Note that the logical drive ID and Target:Lun match up just fine!

 

 

The OS side:

Please compare to what FreeBSD’s mfi driver assigns…

mfid0: 32768MB (67108864 sectors) RAID volume (no label) is optimal
mfid1: 512MB (1048576 sectors) RAID volume (no label) is optimal
mfid2: 16384MB (33554432 sectors) RAID volume (no label) is optimal
mfid3: 65536MB (134217728 sectors) RAID volume (no label) is optimal
mfid4: 645832MB (1322663936 sectors) RAID volume (no label) is optimal
mfid5: 3337472MB (6835142656 sectors) RAID volume 'raid10data' is optimal

At install time it was cute enough to *drums* assign the 3.X T lun as mfid0. So we installed FreeBSD 10 on the LUN that stores my VMs.

That, of course, killed the LVM headers and a few gigabytes of data.

 

My next post will skip over reinstalling to the right lun (identified from live cd system) and instead describe how I went about getting the data back.

 

Part three: Storage migration, disaster recovery and friends


All posts:

What I had not expected was how hard it would be to decide on an actual solution.

 Picking a Hypervisor

For a lab I would need:

  • nested virt
  • high performance
  • low overhead to the same due to power etc.
  • easy cloning of vms and labs
  • flexible networking
  • easy scripting
  • wide storage options and easy migration
  • thin provisioning of some kind

 

If you know all the products and their drawbacks it turned into a constant forth-and-back between the different hypervisors and ecosystems.

 

VMWare:

VMWare always sneaked back due to feature reliability and performance consistency and then got kicked back out for the lack of many features like API and storage migration w/o a full vCenter install.

I knew it would deliver a good (600-900MBish) performance under any circumstance, where i.e. Xen can be all over the place from 150 to 1900MB/s…

Another downside was that in VMWare  my SolarFlare 5122 will definitely never  expose the 256VNICs. And I’d like to have em.

Installing MegaCli in ESXi is also a bit annoying.

On the pro side there’s the Cisco Nexus1000V and many other similar *gasp* appliances.

And, the perfect emulation. No “half” disk drivers. no cheapass BIOS.

In the end, I like to have my stuff licensed and to use the full power of a VMWare setup I’d need to go with vCenter + Enterprise Lic. No fun.

 

XenServer:

Just LOL.

While XenServer has great features for VM Cloning it’s just not my cup of tea. Too much very bad python code. Too many windows-user cludges. Broken networking all over.

Any expectation of storage flexibility would be in vain, needing backporting and recompiling software to the dom0 kernel using their SDK. Definitely not an easy solution if you wanna be able to flip between iSCSI, Infiniband, md and whatever else *looks* interesting. This should be a lab after all, and I don’t see any chance running something like the Storwise VSA in this. Nested ESXi for that, and that’s not on the roadmap for XenServer. If anything still is.

It would probably work best for SolarFlare. I’ll admit that.

 

CloudWeavers:

This is what will run in many VMs, but I don’t wanna break layering, so my underlying hypervisor and solution should not be the same as in the VMs. I am not yet sure if it’s the right decision.

This would be the prime KVM choice since they already deliver a well-tuned configuration.

What worries me is that, while MooseFS’ FUSE client scales good enough on a single hypervisor node, it would end up with a lot of additional context switching / trashing if I use it on the main node and in the clients. There might be smarter ways around this, i.e. by having a fat global pool in the “layer1” hypervisor and using that from the above layers, too. More probably it’d turn into a large disaster 🙂

 

LXC:

Pointless, no hypervisor, one single kernel instance can’t successfully pretend being a bunch of OSDs and clients 🙂

 

Plain Xen:

This is what I already have and went with, especially to make use of tmem and run the Ceph labs as paravirt domUs. This way I know nothing will get in the way performance wise.

There’s one thing you should know though, comparing Xen vs. ESXi or a licensed VMWare though:

Xen’s powermanagement is brokenbrokenbroken:

  • Newer deep-idle CPU states are all unsupported
  • The utility to manage CPU power management is broken as well. Since 4.3 nothing works any more.
  • Even if you free + shutdown a core from dom0 it’ll not be put to sleep

You can definitely tell from the power intake fan speed that Xen, even idle consumes more power than an idle Linux kernel would. Spinning up a PV domU has no impact, spinning up a HVM one is also a noticable increase in fan whoosh.

ESXi is far better integrated so I am expecting like 100 Euro (personal unfunded opinion) per year of additional energy wasted over VMWare.

My choice for Xen is mostly

  • the bleeding edge features like tmem
  • the really crazy stuff like vTPM and whatever of the cool features ain’t broken at any given time.
  • leverage any storage trick I want and have available in a (thanks to Alpine Linux) very recent Linux kernel
  • put in place ZFS, maybe in a dedicated driver domain
  • also be able to use MooseFS and last, but most interesting
  • all the things that never work on other hypervisors – CPU hotplug, dynamic ram changes…
  • storage domUs!!!!!

 

I think in a lab running 20-30 loaded VMs it will be cruicial to optimize in the memory subsystem.

Same goes for having the least possible CPU overhead, under load this will help.

Last, concurrently being able to use different storage techs means I can chose different levels of availability and speed – albeit not _having to_ since there’s a large SSD monster underneath it.

I’m also quite sure the disks will switch from Raid10 to Raid5. They just won’t see any random IO any more.

The “Raid5 is not OK” Disclaimer

Oh, and yes. Just to mention it. I’m aware I’m running green drives behind a controller. I know about Raid5 rebuild times (actually, they’re much lower on HW raid. About 30% of software raid) and the thing is…

If I see disk dropouts (yet to be seen), I’ll replace the dumb thing ASAP. It makes me cringe to read about people considering this a raid controller issue. If the damn disk can’t read a block for so long that the controller drops it out… Then I’m glad I have that controller and it did the right thing.

Such (block errored) disks are nice as media in secondary NAS storage or as doorstops, but not for a raid. Maybe I just hit extremely lucky in having no media errors at all off them? Definitely not what you’d see in a dedicated server at a mass hoster.

I’ve also patched my Check_MK Smart plugin to track the smart stats from the raid PDisks, so anything SMART notices I’ll be immediately be aware of. Why the green disks in the first place? Well – power and noise benefits are huge. If I had some more space I’d consider a Raid6 of 8 of them, but not until I move to a bigger place.

 

Coming up next:

A colleague offered me some company when setting up a final storage layout.

We build a dedicated storage domU with PCI passthrough’ed MegaRaid controller and ZFS. The install had a little issue…

This is what the next posts will be about, one describing how to build a storage domU.

Also, what pitfalls to expect, and then a focus on losing data (sigh) and getting it back.

I’ll close with some lessons learned. 🙂

Part two: Storage migration, disaster recovery and friends


All posts:

 Go and find me a new host. Keep some money for foods.

So, in march and april I set out to build a *home* server that could handle a Ceph lab, and would behave mostly like real hardware. That equates to disks being slow, SSDs being fast, and RAM being, well, actual RAM. Writing to two disks should ideally also not immediately turn into an IO blender because they reside on one (uncached) spindle.

I think ocver all I spent 30 hours on Ebay and in shops to find good hardware for a cheap price.

 

This is what I gathered:

  • Xeon 2680V2 CPU (some ES model) with 8 instead of 10 cores but same 25MB of cache. It’s also overclockable, should I ever not resist that
  • Supermicro  X9SRL-F mainboard. There are better models with SAS and i350 NICs but I wanted to be a little more price-conservative there
  • 8x8GB DDR3 Ram which I recycled from other servers
  • 5x Hitachi SSD400M SSDs – serious business, enterprise SAS SSDs.
  • The old LSI 9260 controller
  • The old WD green disks

The other SSD option had been Samsung SM843T but their seller didn’t want to give out a receipt. I’m really happy I opted for “legit” and ended up with a better deal just a week later:

The Hitachis are like the big brother of the Intel DC S3700 SSD we all love. I had been looking for those on the cheap for like half a year and then hit lucky. At 400GB capacity each it meant I could make good use of VM cloning etc. and generally never look back to moving VMs from one pool to another for space.

 

I had (and still have) a lot of trouble with the power supply. Those intel CPUs take very low power on idle, even at the first stage of the boot. So the PSU, while on the intel HCL, would actually turn off after half a second when you had very few components installed. A hell of a bug to understand since you normally remove components to trace issues.

Why did I do that? oh, because the supermicro ipmi gave errors on some memory module. Which was OK but not fully supported. Supermicro is just too cheap to have good IPMI code.

Meh.

Some benchmarking, using 4(!) SSDs was done and incredibly.

Using my LSI tuning script I was able to hit sustained 1.8GB/s writes and sustained 2.2GB/s reads.

After some more thinking I decided to check out Raid5 which (thanks to the controller using parity to calculate every 4th? block) still gave a 1.8GB/s read 1.2GB/s write.

Completely crazy performance.

To get the full Raid5 speed I had to turn on Adaptive Read Ahead. Otherwise it was around 500MB/s, aka a single SSDs read speed.

One problem that stuck around was that the controller would / will not enable the SSDs write cache, no matter what you tell it!

This is a huge issue considering each of those SSDs has 512MB(ytes) of well-protected cache.

The SSD is on LSIs HCL for this very controller so this is a bit of a bugger. I’ll get back to this in a later post since by now I *have* found something fishy in the controllers’ output that might be the cause.

Nonetheless: Especially in a raid5 scenario this will have a lot of impact on write latency and IOPS.

Oh, generally: this SSD model and latency? not a large concern 🙂

 

Part one: Storage migration, disaster recovery and friends


This is the first post of a series describing recent changes I did, some data loss, recovering from it and evaluating damage.
All posts:

 

Starting point.

I am building a new Xen Host for my home lab. It was supposed to handle one or two full Ceph labs at high load.The old machine just couldn’t do that.

 

What I had was a Core2 Q6600 quadcore CPU on an Intel S3210 board (IPMI, yay). It had 8GB of Ram, a IBM M5015 Raid Controller and Dual Nics. For storage I had a Raid10 over 4x2TB WD Green drives fronted by a Raid0 Flashcache Device build from two Samsung 830’s. Due to the old chipset the SSDs were limited somewhere around 730MB/s read/write speed.

The main problems were lack of CPU instructions (nested paging etc) for advanced or bleeding edge Xen features.

  • Memory overcommit using XenPaging only works if you have a more recent CPU than mine. (Of course this defeats the point since a more recent Xeon can handle enough RAM in the first place. But still)
  • The second thing was that PVH mode for FreeBSD needed a more recent CPU and last,
  • Nested Virt with Xen is getting somewhere which would be interesting for running ESXi or many Cloudweavers instances w/o performance impact

So, I couldn’t have many nice things!

Also I knew the consumer SSDs had too much latency for a highspeed cache.

For Ceph there was the added requirement of handling the Ceph Journals (SSD) for multiple OSDs and not exposing bottlenecks and IO variances from using the same SSD a dozen times.

 

I’m unhappy to replace the server while it was so far never really over 2-3% of average CPU – but since I want to do A LOT more with Ceph and Cloudweavers it was time to take a step forward. I spend some time calculating how far the step could be and  found that I would have to settle somewhere around ~1600 Euro for everything.

Trip to OpenNebulaconf


This year also saw the first ever OpenNebula conference. I was there for a really short time only, since I’d been coming from the open source backup conference at cologne.

Let me say it was a harder, longer trip than I could handle, two conferences in two days is already bad, but if you also need to prepare stuff it gets rough.

So, how was it?

Getting there: the (almost endless) ride

So, an almost sleepless ride to cologne and then another pretty long one to berlin, a short nap, and every free minute spend on the lab (the server failed the final test reboot like 2 hours before my 3am train departed…). A disaster, but at least the people started to be less rude (running into you, etc) the closer I got to berlin.

At some point there was a nice young consultant woman sitting next to me who *also* fought sleep while she frantically worked on some papers. Couldn’t help smiling.

By the time I arrived I had like 37 hours of work/talks/travel versus 3 hours of sleep. You bet I *love* the beds at my fav berlin hotel (park inn alexanderplatz) when I arrive after a ride like that.

I’m in the wrong place and there’s a nazi for breakfast.

The next day started out bad – the hotel was *called*, but not located at, Alexanderplatz. Not fun considering I had to put down a lot of money to get a room at Alexanderplatz, had planned to save some time by being close by to the venue, and that I had a kinda weird cab driver to take me to the other place. Being completely exhausted even in the morning I really didn’t care to hear about the lower amounts of foreign population in East-berlin due to the non-exchangeable nature of the GDR mark.

Last to go

Having arrived I found the conference reception desk, and apparently I was totally the last person to arrive:  the guy at the desk immediately knew who I am. I browsed around a little, immediately caught sight of the super cool inovex opennebula lab (acrylic casing, 8 i5 nodes), then had some coffee and settled for the sofa.

Oh, THERE!

I tried to get my “personal IT” working so I could drop a message to carlo daffara who only had little time left till his flight and at some point I realized the impatient guy around the corner was him, waiting. 🙂

With that sorted we spent almost two hours chatting and I was surprised at some of the stuff they’re doing at cloudweavers. It doesn’t easily happen that you meet anyone up for a discussion of IO queue/latency/bw issues. Like, noone. Less than that if you’re talking about CEOs. Now, there he is and he’s even got real solutions in the works that noone has ever worked on as methodically. And stuff like this is all just a little sidequest for cloudweavers. I’m amazed.

Lunch break? Slides!

So far I had seen no talks but at least got to watch the amazing lightning talks – once they’re online, watch all of them.

I tried to make my slides more useful, fixed bugs in my new opennebula nagios checks and, well, generally panicked.

Then it was time for the talk, and I tried to do well. 🙂

Slides suck!

Next time I’ll stick with 5 slides and just tell what I think – I don’t need that bullshit powerpoint to get people interested so why bother.

I think I managed to have some minds *click* on the idea of monitoring the large scope of an infrastructure instead of just details. One of the key points was to monitor free capacity instead of usage. In a cloud env I think this is a must have.

I didn’t get the time to add a single tiny BI rule for my setup, so I skipped most of the business intelligence part.

One sad/funny point was that I went on forever about fully dynamic configuration, but missed the main point:

This will be a downloadable selfconfiguring monitoring appliance you can get via the marketplace.

I just didn’t remember to say it.

The reception was good anyway and I hope I helped some of the people – not to mention that it was really hard to talk in front of so many of them! I’m still suprised if someone comes to me and says he liked the talk. Some day I’ll stop worrying.

0.25 days of conference left

I watched a few more talks and it was hard to decide which one to look at – for example I went to hear about rOCCI and it was very worth it but missed the talk from BBC. I’m so looking forward to the recordings.

After that talk, there was another break and then the conference ended with a very short speech from the OpenNebula guys. Many people including me just kept sitting, still eager for more talks. Seems there’s room for a 3-day conference if the topic is that interesting 🙂

What else…

I think it was great that there was multiple companies behind hosting the conference, it seemed to open up discussions a little. I was surprised that the NetWays team really held back marketing wise, which is far different from what I heard from (non-MK 🙂 visitors of other (mostly monitoring-) conferences they have a role in. They did an incredibly good job at organizing stuff. It’s hard to describe – I’m used to the utter chaos of CCC and such conferences and what Netways put up is the exact opposite. Everything worked. Everyone I talked to was happy with how smooth the conference went. Really, great work.

After the conference I had some sleep and then went for drinks with the opennebula guys. Sitting outside after a few burgers I had the second “unfun” event of the day when some old unhappy man started to insult, attack and shove around random people of the group. My first thought was just “yeah right, that’s what we get for being in Mitte instead of Kreuzberg”.

Since I was the only german I tried to tell him to stop acting like a 12-year old idiot, but to little success. After some time he finally left. I think this guy was actually just full of self-hate and wanted someone to hit him. Very weird.

How did I make it back?

After this unasked interruption we moved on a few corners and went to CCCP bar, which was still mostly a tourist place, but a lot more the Berlin I’m used to. Good drinks, a lot of opennebula and other chat and a nice bartender(ess) made it very hard to leave.

At 3 or 4 I still somehow started walking back to my hotel. I have no clue how I actually got there.

The next day I got a lot more sleep and instead of getting drunk again I was already adding some more bugfixes to the KVM checks 🙂

Although I missed the OpenNebula team – they’re extremely interesting and nice people.

Final words

I missed some of the best talks, plus the hacking sessions, plus the gettogether. Next year I shall not make that mistake!

Soon I’ll also do a writeup about the technical  bits of the monitoring thingy.

Zyxel NSA 365 Packaging


Small update – a few hours later (so that’s where the evening has gone)

Important links:

You must install “FFP”, an extended package manager

http://zyxel.nas-central.org/wiki/FFP_as_zypkg

I had to do it mostly the manual way, using zypkg -i on the ffp package.

Setting up build env and packaging:

http://forum.nas-central.org/viewtopic.php?f=249&t=6239

I’ve managed to add NFS (official package) and then the build env.

Bacula-SD built after I also added a MySQL package for linking into “bscan”.

I created a package but it still lacks a “start” script – it’s named start, but it seems the standard bacula-ctl-sd will do the job.

Sadly, I should still make a better package for this, then add also add a Check_MK agent package.

 

The performance of the NAS is as good as reported, I’m running a rsync over nfs (async,intr,soft,wsize=32768) with a constant 40-50MB/s. As a backup target this will definitely suffice.

 

 

The fan is not as loud as reported, so I postponed buying a Papst 612 FL fan for it.

I paid 109 Euro since I bought it in a local shop, online prices range as low as 72 Euros.

The build quality is uh… let me put it like this: in accordance with the price.

You unpack it, try to find how to open the door and then… front fell off.

 

One more small update since it’s so incredible:

Running FTP on a file on a USB3 stick attached to the front USB port.
So, I’m downloading from the NSA325 to my fileserver VM, which has a too tiny /dev/shm to fit the whole file.
First, I was quite happy seeing 88MB/s throughput. Then I figured “let’s set it to do 1MB readahead via /sys”.
Look at this:

ncftp /SanDisk-Extreme-00011 > get CentOS-6.2.img.bz2
CentOS-6.2.img.bz2: ETA: 0:04 104.32/577.56 MB 109.75 MB/s Local write failed after 211842400 bytes had been received: No space left on device.

109MB/s – so one 1MB/s per Euro spent 😉
One could consider turning on Jumbo frames, but at that speed, who would I be to not be content?

Zyxel NSA-325 NAS / Media Server


I just went shopping and besides some proper going-anywhere shoes from new rock I finally bought a home NAS box. The model is a Zyxel NSA-325 which is only a little less powerful as i.e. a Synology DS213+ or DS413 but only costs 1/3 resp. 1/4 of the money those cost.
Being me, it’s not entirely unreasonable I’ll want to run more than one of whichever NAS storage I get.

The downside of the Zyxel NAS is it’s avverage webinterface and limited OS.
At work I chatted with $customers about this and found I actually just want something that lets me edit /etc/exports and maybe run targetcli, definitely a GUI isn’t high-ranking with me.

It seems these run Archlinux but there’s some hickups. After my underground ride home, I’m quite sure I won’t “update” to Arch as a priority.

Why no ArchLinux?

Now, it’s like that:

The NAS will be used to store backups, I’m quite sure I can hack a Bacula Storage Daemon (SD) package for it’s native format.
The bacula Director will be running on my “home cluster” that spans the Nexus7 tablet and the raspberry Pi.
By running the SD on the NAS I’ll make sure that only metadata traffic is directed to those tiny powerless cluster nodes.
By taking the backups off my home server I’ll stop wasting raid10 space on it for storing *backups*.

While I was at it, I also grabbed two 2TB Caviar green disks.

So this is the plan:


Put a 1.5TB random disk into the NAS. (Not the greens, you’ll see why)
Flip-rotate the two new 2TB green for the two 1.5TB ones in my server, so it’ll finally have 2TB disks only.
Online resize, tadaaaa, increase to 4TB usable space. Taking out 600GB of backup storage means I’m looking at 1.6TB free Raid10 space. 🙂

While at it, add a 2.5″ drive cage to the server and add 2 Samsung 530’s.
Replace the current Perc5i Raid controller in the server with one that has SAS6G/SATA3 ports (ibm5015 off ebay)
This will also allow for better placing and ventilation in the server since the perc5 didn’t really fit in my case.
Since I don’t have the “Performance Accelerator Key” (fucking $400 hardware dongle) for the controller, I can’t use LSI CacheCade and need to settle for something OS-based.

That means I shall upgrade the Xen host to AlpineLinux from OracleVM2 it currently runs.

Then I can use Flashcache (or possibly bcache but I don’t like it) to enable read-caching via the SSDs.
And since it’s just read-caching there’s nothing bad about running them as a raid0 for *cough* added performance. After that, I doubt I’ll ever remember this box is backed by cheap and silent, but foremost *cool* WD green drives.

Then, a not so fun step, some pisses-me-off-already chroot building so I can use the goddamned MegaCLI to monitor the raid controller on AlpineLinux.

Finally, put the two now-free 1.5TB WD green in the NAS box.
I plan to also put the cobbler distro mirror on it and those ISOS that are easily obtained.

End result:

  1. Replace disks and controller, add SSDs => A lot more performance and space in the home server.
  2. Able to fully use TMEM on the home server => more free RAM, thus longer lifecycle for the server.
  3. dedicated storage for backups and cobbler => *everything* infrastructure can run on the few-Watts only RaspiNexus cluster.
  4. A shitload of storage migrations, HW replacements etc.  => I totally don’t want to bother with replacing the OS on the NAS.