Help troubleshoot 10gb nic slowness

TeleFragger

[H]ard|Gawd
Joined
Nov 10, 2005
Messages
1,083
Ok so I have an old thread where I used old Mellanox CX1 cards. they have a proprietary port called CX4...
well in this thread, i talk about it and get it working..
https://hardforum.com/threads/starting-to-play-with-10gb-mellanox-and-need-help.1973101/

so in all have been great but just last few weeks, i have had tons of issues with gaming slowness.

I went to speedtest.net and am getting 525mb down BUT .03 up.. .8 up.. 1.0mb up...

I swapped the card out, swapped cable out and still issues. Compared settings to my plex box that also has the same card in it and still nothing...

So I reinstalled Windows 10 and nope.. still issues.

Another weird part is while i can navigate the web, you dont notice the issues.. I can game (warzone - rebirth), that is where I start seeing the lagginess....

I tried getting to my file server that has the same card in it and explorer wont show anything, so that was another weird thing for me to note...

I just dont know what to do or try....

I have an hp procurve 6400cl that my 10gb nics connect too, then they go to a brocade fastiron fcx648s that then goes to my xfinity cable modem and out..

my plex box goes out and speedtest.net is 525mb down and 18mb up... the 18mb up that I expect....
thus the issue is not my switches or internet as this box works as expected...

From my plex to my server via iperf3 I get...

C:\Users\Jeff\Desktop\iperf-3.1.3-win64>iperf3.exe -c riggsfamily
Connecting to host riggsfamily, port 5201
[ 4] local 10.66.10.24 port 51235 connected to 10.66.10.66 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 982 MBytes 8.23 Gbits/sec
[ 4] 1.00-2.00 sec 1.04 GBytes 8.90 Gbits/sec
[ 4] 2.00-3.00 sec 1.00 GBytes 8.60 Gbits/sec
[ 4] 3.00-4.00 sec 968 MBytes 8.12 Gbits/sec
[ 4] 4.00-5.00 sec 888 MBytes 7.45 Gbits/sec
[ 4] 5.00-6.00 sec 1.05 GBytes 9.05 Gbits/sec
[ 4] 6.00-7.00 sec 1.05 GBytes 8.98 Gbits/sec
[ 4] 7.00-8.00 sec 1.05 GBytes 8.98 Gbits/sec
[ 4] 8.00-9.00 sec 1.05 GBytes 9.05 Gbits/sec
[ 4] 9.00-10.00 sec 1.05 GBytes 9.00 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 10.1 GBytes 8.64 Gbits/sec sender
[ 4] 0.00-10.00 sec 10.1 GBytes 8.64 Gbits/sec receiver

iperf Done.



from my gaming rig, I do see an issue initially with iperf3....
not sure what is causing it...

C:\Users\Jeff\Desktop\iperf-3.1.3-win64>iperf3 -c riggsfamily
Connecting to host riggsfamily, port 5201
[ 4] local 10.66.10.51 port 61608 connected to 10.66.10.66 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 256 KBytes 2.07 Mbits/sec
[ 4] 1.01-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.01-3.01 sec 396 MBytes 3.30 Gbits/sec
[ 4] 3.01-4.00 sec 1015 MBytes 8.63 Gbits/sec
[ 4] 4.00-5.00 sec 1.00 GBytes 8.62 Gbits/sec
[ 4] 5.00-6.00 sec 1.00 GBytes 8.63 Gbits/sec
[ 4] 6.00-7.00 sec 1.01 GBytes 8.65 Gbits/sec
[ 4] 7.00-8.00 sec 1018 MBytes 8.54 Gbits/sec
[ 4] 8.00-9.00 sec 1.01 GBytes 8.65 Gbits/sec
[ 4] 9.00-10.00 sec 864 MBytes 7.25 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 7.24 GBytes 6.22 Gbits/sec sender
[ 4] 0.00-10.00 sec 7.24 GBytes 6.22 Gbits/sec receiver

iperf Done.


let me know your thoughts please....
 
I haven't had a chance to play with Mellanox cards, but this feels like maybe an MTU issue or maybe a TCP offloading issue.

I'd try disabling TCP segmentation offloading in the advanced driver options and see if that helps. If not, MTU troubleshooting is a bit tricky, not sure if there's a good reference, but this might get you started: https://techcommunity.microsoft.com...-connectivity-issues-with-windows/ba-p/257636
 
Try different port on the switch.
tried that.. on the hp procurve...

1 of the 10gb ports on the procurve goes to the brocade.. but as said plex is on procurve and goes through brocade as expected....

will still look at trying to swap with plex port to see... just in case...
 
I haven't had a chance to play with Mellanox cards, but this feels like maybe an MTU issue or maybe a TCP offloading issue.

I'd try disabling TCP segmentation offloading in the advanced driver options and see if that helps. If not, MTU troubleshooting is a bit tricky, not sure if there's a good reference, but this might get you started: https://techcommunity.microsoft.com...-connectivity-issues-with-windows/ba-p/257636

Thank you!!!!!

following that link and resetting my mtu got me to now be able to get to my server via unc and can access it... still getting 500+mb down and upload is going between 3mb and 5mb.. better.. but not the 18mb i am getting on my plex server or my 1gb line... more to look into
 
Which NIC? Did you set MTU there too? All devices should be same MTU.. but you prolly know that.

Have you tried another tool other than iperf3.x? Try 2.x? Try a "-b" to set BW.

When I was testing 25G/40G/100G at Intel in linux land we used various tools to measure speeds. Some were not accurate. :( + flaky drivers and bad firmware.

Don't have my notes atm as I'm traveling....
 
Sorry ADHD kicking in and i am all over the place... so my apologies for that

I set out to get a new baseline...

figured let me go to my plex box and go to speedtest.net (windows 10 box) and low and behold it too is showing slowness!!!!
so now it is seeming like it is internal towards my switch...
well trying to figure out where at now..

from plex to server
iperf3 -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 10.3 GBytes 8.83 Gbits/sec sender
[ 4] 0.00-10.00 sec 10.3 GBytes 8.83 Gbits/sec receiver

but speedtest.net
PING ms DOWNLOAD Mbps UPLOAD Mbps
15 502.55 1.41



i noticed all of this on my gaming rig which had 2 problems..
1. couldnt get to server share (can now after MTU fix.. thx toast0
2. slow speetest.net results... to which seems both are doing now..

iperf 3 - not the best but mtu seems to made it a bit better
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 4.40 GBytes 3.77 Gbits/sec sender
[ 4] 0.00-10.01 sec 4.40 GBytes 3.77 Gbits/sec receiver

speedtest.net
PING ms DOWNLOAD Mbps UPLOAD Mbps
15 525.95 8.27

gaming rig on 1gb line (broade switch and not touching 10gb hp procurve)
PING ms DOWNLOAD Mbps UPLOAD Mbps
15 525.36 18.01


Already reboot hp procurve... but gonna reboot it ALL..
procurve, brocade, try different cable between procurve and brocade and even try 2nd port of brocade...
 
Which NIC? Did you set MTU there too? All devices should be same MTU.. but you prolly know that.

Have you tried another tool other than iperf3.x? Try 2.x? Try a "-b" to set BW.

When I was testing 25G/40G/100G at Intel in linux land we used various tools to measure speeds. Some were not accurate. :( + flaky drivers and bad firmware.

Don't have my notes atm as I'm traveling....

card 11 was not 1500...
C:\Users\Jeff\Desktop\iperf-3.1.3-win64>netsh int ipv4 show int

Idx Met MTU State Name
--- ---------- ---------- ------------ ---------------------------
1 75 4294967295 connected Loopback Pseudo-Interface 1
10 5 1500 disconnected Ethernet 2
11 15 2034 connected 10GB
18 5 1500 disconnected Slot01 x8 2


so i ran the command and now it is... and here is what we got..

C:\Windows\system32>netsh int ipv4 show int

Idx Met MTU State Name
--- ---------- ---------- ------------ ---------------------------
1 75 4294967295 connected Loopback Pseudo-Interface 1
10 5 1500 disconnected Ethernet 2
11 15 1500 connected 10GB
18 5 1500 disconnected Slot01 x8 2
 
oh and i truly don't care about the #'s i get.. i just see gaming lagginess.. I already lag due to my gtx 1060... so i dont need any other helpers while trying to play rebirth!!!!!!!!! LOL
 
Really seems like a brocade issue. Try different port on the switch? Reboot the brocade for sure
 
Have you isolated it to the NIC? What do you get with the 1GbE port? Typically throughput has little impact with latency. I've ran 10 GbE SFP+ card with TwinAx and saw no difference with internet latency since upstream devices (router, modem, ISP) will always be the bottleneck. I do notice better ping with IPv6 enabled (if supported from end-to-end) but that's probably packets because are travelling using more efficient routes with newer equipment.

Also iperf numbers can be a little weird under Windows, runs good on Linux but I believe you need multiple parallel streams using a certain flag to get an accurate measurement.
 
Is it possible to temporarily bypass the switch and go more directly out to the internet? Even a basic 1gbe port should be able to handle the ~20mbps youre expecting from upload. Also, since were testing MTUs, are jumbo frames enabled on your switch? Whats the frame size? You should make sure thats also set to 1500 as well.
 
Back
Top