Last night I was running some more benchmarks to verify the infiniband links are stable and checking if there is any negative impact if you add a 4xSDR (10Gbit) node to the other 4xDDR (20Gbit) nodes.
I was mostly looking at the rdma bandwidth with connected mode, as this is what should apply to glusterFS. I normally turned off the firewall, but I think technically it doesn’t matter with RDMA.
I noticed when changing the iterations in ib_rdma_bw to 200000 the bandwidth average displayed would drop from 2.8GB/s down to 40-50MB/s.
What had happened?
I decided to run multiple tests over all the connections (A to B, B to C, A to C, …) and found the error kept coming up once I ran a longer test than the default.
After that it was either a bug in ib_rdma_bw or my switch. I found it unlikely to be the switch, at those data rates an error should show almost immediately, like within 20GB, not after a few 100)
Turns out there was an overflow in ib_rdma_bw. Problem solved 🙂
That’s a older picture of the Cisco SFS3504 switch / gateway:
Right now there’s 3 infiniband cables going in and no gigabit cable coming out.
I disabled the gateway function until I know how to use infiniband partitions correcty. They’ll be mapped on ethernet VLANs so that non-IB hosts can access them via IPoIB, too. But when I just enabled this without using VLANs etc. it meant that the IB hosts would see their own and forgeign IPs twice – via IB and Ethernet. A lot of chaos resulted 🙂