Myricom 10GbE performance on OS X 10.8

[nry]

I have a small 10GbE network at home (used for work) consisting of a NAS and a bunch of ESXi hosts.
I recently decided to take the plunge and spend far too much money getting my iMac (i5 / 16GB DDR3 / SSD) onto this network using a Sonnet Echo Express thunderbolt enclosure, Myricom 10G-PCIE2-8B2-2S, and some 10Gb SFP+ modules.

Was extremely painless to install and had it all running very quickly, but I seem to be hitting a limit of 4.37Gbit/s and while this is probably more than adequate for my workstation , I would like to be able to get slightly better performance considering the price I paid.

I am 100% the DAC cables, SFP+ modules and fiber cables are not the issue here as I have tested them in other NICs and all get performance very similar to the below

iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[  3] local 10.0.21.123 port 50466 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.39 Gbits/sec

Next the iperf tests on the iMac....

iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 10.0.21.124 port 53614 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.09 GBytes  4.37 Gbits/sec

Read somewhere changing the tcp window scale option can improve things, not that I have any clue what this is, bug figure I'd give it a go as by default it is on

➜  ~  sysctl net.inet.tcp.rfc1323
net.inet.tcp.rfc1323: 1
➜  ~  sudo sysctl -w net.inet.tcp.rfc1323=0
net.inet.tcp.rfc1323: 1 -> 0

And the results, well that didn't do much good

➜  ~  iperf -c 10.0.21.102
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 10.0.21.124 port 53661 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.32 GBytes  1.99 Gbits/sec

I have followed every tip inside the Myricom readme here https://www.myricom.com/scs/READMES/README.myri10ge-macosx
And a whole range of tips/changes here https://www.myricom.com/software/my...hoot-slow-myri10ge-or-mx-10g-performance.html

Also tried the suggestions for the iperf options...

  ~  iperf -c 10.0.21.102 -w 768k -l 256k
------------------------------------------------------------
Client connecting to 10.0.21.102, TCP port 5001
TCP window size:  769 KByte (WARNING: requested  768 KByte)
------------------------------------------------------------
[  4] local 10.0.21.124 port 54258 connected with 10.0.21.102 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.04 GBytes  4.33 Gbits/sec

But still around the 4.37Gbit/s limit, starting to think maybe the thunderbolt link is limiting it somehow or the Sonnet enclosure isn't good enough. I think the card only runs at PCIe x4 mode in this enclosure but that should give it enough bandwidth to max out 10GbE

On the Myricom readme they provide a way to show the PCIe bandwidth...

➜  ~  sysctl net.myri10ge | grep dma
net.myri10ge.en3.dma_read_bw_MBs: 806
net.myri10ge.en3.dma_write_bw_MBs: 825
net.myri10ge.en3.dma_read_write_bw_MBs: 978
net.myri10ge.en4.dma_read_bw_MBs: 805
net.myri10ge.en4.dma_write_bw_MBs: 825
net.myri10ge.en4.dma_read_write_bw_MBs: 978

I seem to be limited to 540MBs which I'm sure I could live with but would like to max this device out if possible!

Today I may try test the card inside a Ubuntu box to see how it performs, just to double check the NIC is able to perform.

Any suggestions on changes I could try to improve this?

 

[Phog]

I think the card only runs at PCIe x4 mode in this enclosure

This is my guess. Even though it theoretically has the capability, doesn't mean you will get it. I get half speed on my 10GbE Myricom's if they are in x4 ports.

 

[nry]

Yes I think it does only run at x4, to quote Sonnets site

The PCIe slot will physically accommodate up to a x8 PCIe card, however the actual electrical bandwidth of Thunderbolt is x4.

I just figured since PCIe 2.0 is 500MB/s it should be capable of maxing out the thunderbolt connection

(from wikipedia)

Capacity

Per lane (each direction):

v1.x: 250 MB/s (2.5 GT/s)
v2.x: 500 MB/s (5 GT/s)
v3.0: 985 MB/s (8 GT/s)
v4.0: 1969 MB/s (16 GT/s)