This is the first post of a series describing recent changes I did, some data loss, recovering from it and evaluating damage.
I am building a new Xen Host for my home lab. It was supposed to handle one or two full Ceph labs at high load.The old machine just couldn’t do that.
What I had was a Core2 Q6600 quadcore CPU on an Intel S3210 board (IPMI, yay). It had 8GB of Ram, a IBM M5015 Raid Controller and Dual Nics. For storage I had a Raid10 over 4x2TB WD Green drives fronted by a Raid0 Flashcache Device build from two Samsung 830’s. Due to the old chipset the SSDs were limited somewhere around 730MB/s read/write speed.
The main problems were lack of CPU instructions (nested paging etc) for advanced or bleeding edge Xen features.
- Memory overcommit using XenPaging only works if you have a more recent CPU than mine. (Of course this defeats the point since a more recent Xeon can handle enough RAM in the first place. But still)
- The second thing was that PVH mode for FreeBSD needed a more recent CPU and last,
- Nested Virt with Xen is getting somewhere which would be interesting for running ESXi or many Cloudweavers instances w/o performance impact
So, I couldn’t have many nice things!
Also I knew the consumer SSDs had too much latency for a highspeed cache.
For Ceph there was the added requirement of handling the Ceph Journals (SSD) for multiple OSDs and not exposing bottlenecks and IO variances from using the same SSD a dozen times.
I’m unhappy to replace the server while it was so far never really over 2-3% of average CPU – but since I want to do A LOT more with Ceph and Cloudweavers it was time to take a step forward. I spend some time calculating how far the step could be and found that I would have to settle somewhere around ~1600 Euro for everything.