Cisco SFS 3504 – Tracking Ethernet Performance


I successfully set up the infiniband to ethernet gateway after some poking.

dynamic LACP didnt work (no padu received on my huawei switch, so I went back to a 1port config.

I had to find I was only getting around 20MB/s bandwidth using netcat
to dd from an ethernet host to the one attached via IB.

I had seen (untuned!) >340MB/s IB RDMA speed between the two hosts, so I had started out expecting some more. Also throughput between two ethernet host was 80MB/s for the same test.

I decided to start by using a “real” benchmarking tool.

[root@waxh0004 tmp]# iperf -c 192.168.100.105 -u -b 10000m -i 1
————————————————————
Client connecting to 192.168.100.105, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 107 KByte (default)
————————————————————
[ 3] local 192.168.100.107 port 50266 connected with 192.168.100.105 port 5001
[ 3] 0.0- 1.0 sec 61.7 MBytes 518 Mbits/sec
[ 3] 1.0- 2.0 sec 61.0 MBytes 512 Mbits/sec
[ 3] 2.0- 3.0 sec 61.3 MBytes 514 Mbits/sec
[ 3] 3.0- 4.0 sec 61.0 MBytes 512 Mbits/sec
[ 3] 4.0- 5.0 sec 61.3 MBytes 514 Mbits/sec
[ 3] 5.0- 6.0 sec 61.1 MBytes 512 Mbits/sec
[ 3] 6.0- 7.0 sec 61.3 MBytes 514 Mbits/sec
[ 3] 7.0- 8.0 sec 61.0 MBytes 512 Mbits/sec
[ 3] 8.0- 9.0 sec 61.3 MBytes 514 Mbits/sec
[ 3] 0.0-10.0 sec 612 MBytes 513 Mbits/sec
[ 3] Sent 436505 datagrams
[ 3] Server Report:
[ 3] 0.0-10.2 sec 196 MBytes 161 Mbits/sec 15.574 ms 296595/436503 (68%)
[ 3] 0.0-10.2 sec 1 datagrams received out-of-order

Using 2 Threads I can get up to 567mbit/s, this incidicates a cpu bottleneck on the host might be part of the issue.

To make things worse I noticed that I also only get a max speed of 520mbit/s when running iperf between two hosts on the GigE side.

[root@waxh0004 ~]# iperf -c 192.168.100.106 -u -b 1000M -i 1
————————————————————
Client connecting to 192.168.100.106, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 107 KByte (default)
————————————————————
[ 3] local 192.168.100.107 port 46780 connected with 192.168.100.106 port 5001
[ 3] 0.0- 1.0 sec 62.0 MBytes 520 Mbits/sec
[ 3] 1.0- 2.0 sec 60.7 MBytes 509 Mbits/sec
[ 3] 2.0- 3.0 sec 61.6 MBytes 517 Mbits/sec
[ 3] 3.0- 4.0 sec 60.7 MBytes 509 Mbits/sec
[ 3] 4.0- 5.0 sec 61.6 MBytes 517 Mbits/sec
[ 3] 5.0- 6.0 sec 60.7 MBytes 509 Mbits/sec
[ 3] 6.0- 7.0 sec 61.6 MBytes 517 Mbits/sec
[ 3] 7.0- 8.0 sec 60.7 MBytes 509 Mbits/sec
[ 3] 8.0- 9.0 sec 61.6 MBytes 517 Mbits/sec
[ 3] 9.0-10.0 sec 60.7 MBytes 509 Mbits/sec
[ 3] 0.0-10.0 sec 612 MBytes 513 Mbits/sec

Using two threads got it up to 675mbit/s.

I really don’t know what to make of this.

Why is iperf (udp, even using multiple threads) slower than piping dd through netcat via ethernet?

Why don’t I see the same performance drop as with tcp or dd through the gateway+infiniband?

Isn’t GigE supposed to scale to line rate a little easier these days now?
Shouldn’t one be seeing something closer to 100MB/s even w/o using Jumbo frames?

It’s not like the hardware I put in use is low-end…

Next step will be retesting via an Infiniband back-to-back link w/o the gateway and then replacing the infiniband-attached server by a faster one. And the iptables firewall on the test “client” will also need a second look.

edit:
Thanks to teq on #freebsd.de for pointing out to me that 650MBit/s actually equals the 80MB/s I had measured with dd.

Sometimes I just need help for seeing the obvious.

Now it’s time to go up to line rate

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s