Lost BNX2 Broadcom BCM5708 drivers after Ubuntu upgrade


Hey everyone,

this feels so important I’d rather leave a post here to save you the same troubles.

Networking nightmare:

On Ubuntu 14 LTS you’ll need not just the non-free firmware package, but also this last one called:

“linux-image-extra-3.13.0-63-generic 3.13.0-63.103                        amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP”

Otherwise you’ll not have much more than the stock e1000e around for networking, meaning your servers may miss some nics. This was extremely hard to figure due to the fact that *first* my /lib/modules/3.13.0-63 included the bnx2 and bnx2x modules. Amusingly I found it would still boot 3.13.0-61. After the install of the -63 kernel, the modules were gone. It seems there’s some stupid trimming hook.

Installing the linux-image-extra package made the module stick and I have all 4 nics back.

Lost monitoring site:

A really nice feat is how OMD integrates with most distros by just having a zzz-omd.conf that includes the per-site config files. Now, funny enough, this has been in /etc/apache2/conf.d for years. Ubuntu 14.04 doesn’t read that anymore, it only handles /etc/apache2/conf-enabled. Which is more aligned with the Debian way of things (not that I enjoy it, but at least it’s consistent), but HELL why do you need to suddenly change it after you already fucked up?

I was looking for proxy module issues / misconfiguration for ages until I decided to just add random crap to config files and see if it would break Apache. No, it didn’t. After that some greps verified conf.d isn’t read any longer. It’s beyond me why they don’t at least move the contents over.

SAN nightmare:

One more thing that caused the server to not even boot:

Ubuntu has no concept of loading local disk drivers before SAN drivers. It scanned a lot of SAN luns, hit udev rule bugs by running a bad inquiry on each of them and then finally hit the root device mount timeout.

Root was a local SAS attached SATA disk.

Drivers in this case were mptsas and mptfc. You get the idea, yes? Alphabetically “FC” comes before “SAS”. And no, I don’t think Ubuntu is commonly used with SAN boot plus local disk……..

I’m pretty sure once the devs notice the issue they’l go with a highly professional solution like in /etc/grub.d, i.e. 10-mptsas and 40-mptfc. So clever 🙂

Anyway, to sort that out:

Blacklist the mptfc module in /etc/modules.d/blacklist.conf

fire up update-initramfs -u

Load it again from /etc/rc.local. Of course this also means you can’t really use your SAN luns anymore.

no, there’s no hook in their ramdisk framework to change the order. I don’t know what else to do about it.

If it weren’t for the udev issue, adding a script for the modprobe in init-bottom might work (prior to lvm and multipath).

But also, I have not unlimited time in my life to properly fix shit like this. I have searched for a few hours and have not found anything that comes close to a clean solution for the udev or init order problem. And it’s not my box, so even a update-grub hook etc. just wouldn’t cut it.

If my friend whose server this is needs a more long-term fix, it would be to use a HBA from a vendor with an initial higher than “M”, so in that case, switch to QLogic.

In summary, I think this story covers all there is to say about working with Ubuntu 14 on a server.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s