New infiniband benchmarks – ib_rdma_bw overflow :)

Last night I was running some more benchmarks to verify the infiniband links are stable and checking if there is any negative impact if you add a 4xSDR (10Gbit) node to the other 4xDDR (20Gbit) nodes.

I was mostly looking at the rdma bandwidth with connected mode, as this is what should apply to glusterFS. I normally turned off the firewall, but I think technically it doesn’t matter with RDMA.

I noticed when changing the iterations in ib_rdma_bw to 200000 the bandwidth average displayed would drop from 2.8GB/s down to 40-50MB/s.
What had happened?
I decided to run multiple tests over all the connections (A to B, B to C, A to C, …) and found the error kept coming up once I ran a longer test than the default.

After that it was either a bug in ib_rdma_bw or my switch. I found it unlikely to be the switch, at those data rates an error should show almost immediately, like within 20GB, not after a few 100)

Turns out there was an overflow in ib_rdma_bw. Problem solved 🙂

That’s a older picture of the Cisco SFS3504 switch / gateway:
Cisco SFS 3504 infiniband gateway
Right now there’s 3 infiniband cables going in and no gigabit cable coming out.
I disabled the gateway function until I know how to use infiniband partitions correcty. They’ll be mapped on ethernet VLANs so that non-IB hosts can access them via IPoIB, too. But when I just enabled this without using VLANs etc. it meant that the IB hosts would see their own and forgeign IPs twice – via IB and Ethernet. A lot of chaos resulted 🙂


One thought on “New infiniband benchmarks – ib_rdma_bw overflow :)

  1. Pingback: Tweets that mention New infiniband benchmarks – ib_rdma_bw overflow :) « --

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s