Please help troubleshoot Gigabit performance issues...

S

spectrumbx

[H]ard|Gawd
Joined
Apr 2, 2003
Messages
1,647
Hardware:
- Switch: Dell Powerconnect 2716 (jumbo frames enabled)
- Server: 2x Broadcom Gigabit ethernets (bounded - LACP)
- Laptop: Intel PRO/1000 PL Gigabit NIC

Here are my iPerf results:
Code: 
bin/iperf.exe -c 192.174.149.150 -P 1 -i 1 -p 5001 -f m -t 20
------------------------------------------------------------
Client connecting to 192.174.149.150, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[1912] local 192.174.149.161 port 2214 connected with 192.174.149.150 port 5001
[ ID] Interval       Transfer     Bandwidth
[1912]  0.0- 1.0 sec  43.8 MBytes   368 Mbits/sec
[1912]  1.0- 2.0 sec  48.8 MBytes   409 Mbits/sec
[1912]  2.0- 3.0 sec  48.0 MBytes   403 Mbits/sec
[1912]  3.0- 4.0 sec  24.6 MBytes   207 Mbits/sec
[1912]  4.0- 5.0 sec  23.3 MBytes   196 Mbits/sec
[1912]  5.0- 6.0 sec  22.5 MBytes   189 Mbits/sec
[1912]  6.0- 7.0 sec  44.2 MBytes   371 Mbits/sec
[1912]  7.0- 8.0 sec  46.9 MBytes   394 Mbits/sec
[1912]  8.0- 9.0 sec  47.2 MBytes   396 Mbits/sec
[1912]  9.0-10.0 sec  24.3 MBytes   204 Mbits/sec
[1912] 10.0-11.0 sec  22.9 MBytes   192 Mbits/sec
[1912] 11.0-12.0 sec  23.1 MBytes   194 Mbits/sec
[1912] 12.0-13.0 sec  38.3 MBytes   321 Mbits/sec
[1912] 13.0-14.0 sec  46.5 MBytes   390 Mbits/sec
[1912] 14.0-15.0 sec  48.3 MBytes   405 Mbits/sec
[1912] 15.0-16.0 sec  43.4 MBytes   364 Mbits/sec
[1912] 16.0-17.0 sec  46.6 MBytes   391 Mbits/sec
[1912] 17.0-18.0 sec  47.7 MBytes   400 Mbits/sec
[1912] 18.0-19.0 sec  46.3 MBytes   389 Mbits/sec
[1912] 19.0-20.0 sec  46.4 MBytes   389 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[1912]  0.0-20.0 sec   783 MBytes   328 Mbits/sec
Done.

File transfers between the laptop and server are slow and peak at 20-25% network utilization in task manager.
The broadcom NIC management software shows no dropped packet or any other issue.

Why is that I am not seeing better results with my configuration?

Thanks.
 
That looks normal to me. I rarely see gige do more than 50mbyte/sec in real life, and thats with raid drives.
 
I just ran ipef between two computers and my Procurve 1800 switch.

with the same command line parameters as the OP I get only 135 Mbits/sec

yet daily file transfer can sustain 80megabytes/second easy... (numbers from task manager)

So as LittleMe mentioned, might have something to do with IPerf parameters.
 
LittleMe said:
Increase the window size in iperf and test it again.
Click to expand...

Alright, increasing the windows size did improve things a bit.
Code: 
bin/iperf.exe -c 192.174.149.150 -P 1 -i 1 -p 5001 -w 124K -f m -t 20
------------------------------------------------------------
Client connecting to 192.174.149.150, TCP port 5001
TCP window size: 0.12 MByte
------------------------------------------------------------
[3960] local 192.174.149.161 port 3130 connected with 192.174.149.150 port 5001
[ ID] Interval       Transfer     Bandwidth
[3960]  0.0- 1.0 sec  49.1 MBytes   412 Mbits/sec
[3960]  1.0- 2.0 sec  48.8 MBytes   410 Mbits/sec
[3960]  2.0- 3.0 sec  53.5 MBytes   449 Mbits/sec
[3960]  3.0- 4.0 sec  56.6 MBytes   475 Mbits/sec
[3960]  4.0- 5.0 sec  57.6 MBytes   483 Mbits/sec
[3960]  5.0- 6.0 sec  58.8 MBytes   493 Mbits/sec
[3960]  6.0- 7.0 sec  54.3 MBytes   455 Mbits/sec
[3960]  7.0- 8.0 sec  56.9 MBytes   477 Mbits/sec
[3960]  8.0- 9.0 sec  55.8 MBytes   468 Mbits/sec
[3960]  9.0-10.0 sec  57.4 MBytes   481 Mbits/sec
[3960] 10.0-11.0 sec  57.7 MBytes   484 Mbits/sec
[3960] 11.0-12.0 sec  57.8 MBytes   485 Mbits/sec
[3960] 12.0-13.0 sec  59.9 MBytes   502 Mbits/sec
[3960] 13.0-14.0 sec  56.2 MBytes   472 Mbits/sec
[3960] 14.0-15.0 sec  58.3 MBytes   489 Mbits/sec
[3960] 15.0-16.0 sec  60.0 MBytes   504 Mbits/sec
[3960] 16.0-17.0 sec  56.9 MBytes   478 Mbits/sec
[3960] 17.0-18.0 sec  57.1 MBytes   479 Mbits/sec
[3960] 18.0-19.0 sec  58.7 MBytes   492 Mbits/sec
[3960] 19.0-20.0 sec  60.4 MBytes   506 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[3960]  0.0-20.0 sec  1132 MBytes   474 Mbits/sec
Done.

According to this calculator: http://www.kehlet.cx/docs/tcpwin.php?rtt=1&bw=1000, my windows size should be around 124KB.

Druneau said:
I just ran ipef between two computers and my Procurve 1800 switch.

with the same command line parameters as the OP I get only 135 Mbits/sec

yet daily file transfer can sustain 80megabytes/second easy... (numbers from task manager)

So as LittleMe mentioned, might have something to do with IPerf parameters.
Click to expand...

For me, it is quite the opposite.
My real life transfers are lower than the iPerf result.
 
iperf.exe -c 192.168.1.2 -P 1 -i 1 -p 5001 -f m -t 20 -w 65536
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 0.06 MByte
------------------------------------------------------------
[108] local 192.168.1.50 port 54831 connected with 192.168.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[108] 0.0- 1.0 sec 89.9 MBytes 754 Mbits/sec
[108] 1.0- 2.0 sec 93.3 MBytes 782 Mbits/sec
[108] 2.0- 3.0 sec 93.3 MBytes 782 Mbits/sec
[108] 3.0- 4.0 sec 93.3 MBytes 782 Mbits/sec
[108] 4.0- 5.0 sec 93.3 MBytes 783 Mbits/sec
[108] 5.0- 6.0 sec 93.3 MBytes 783 Mbits/sec
[108] 6.0- 7.0 sec 93.3 MBytes 782 Mbits/sec
[108] 7.0- 8.0 sec 93.3 MBytes 783 Mbits/sec
[108] 8.0- 9.0 sec 93.3 MBytes 782 Mbits/sec
[108] 9.0-10.0 sec 92.9 MBytes 779 Mbits/sec
[108] 10.0-11.0 sec 93.3 MBytes 782 Mbits/sec
[108] 11.0-12.0 sec 93.3 MBytes 783 Mbits/sec
[108] 12.0-13.0 sec 93.2 MBytes 782 Mbits/sec
[108] 13.0-14.0 sec 93.3 MBytes 783 Mbits/sec
[108] 14.0-15.0 sec 93.3 MBytes 783 Mbits/sec
[108] 15.0-16.0 sec 93.3 MBytes 783 Mbits/sec
[108] 16.0-17.0 sec 93.3 MBytes 782 Mbits/sec
[108] 17.0-18.0 sec 93.3 MBytes 782 Mbits/sec
[108] 18.0-19.0 sec 93.3 MBytes 782 Mbits/sec
[108] 19.0-20.0 sec 92.8 MBytes 779 Mbits/sec
[ ID] Interval Transfer Bandwidth
[108] 0.0-20.0 sec 1861 MBytes 781 Mbits/sec

Real life, I get about 85MB/s when I copy a file.


This was just for fun, it's between my 2 servers with 10GbE here at home.
iperf -c 192.168.1.50 -w 2M
------------------------------------------------------------
Client connecting to 192.168.1.50, TCP port 5001
TCP window size: 2.00 MByte
------------------------------------------------------------
[108] local 192.168.1.51 port 54849 connected with 192.168.1.50 port 5001
[ ID] Interval Transfer Bandwidth
[108] 0.0-10.0 sec 11.51 GBytes 9.89 Gbits/sec
 
Update
It looks like my laptop has a lot to do with the low transfer.

CPU usage on the server is zero, but my client gets pegged CPU wise during the network activities (iPerf tests).
When running my laptop on battery, my throughput dropped to a bout ~300MBits/s.

With the laptop plugged into the AC, here are my latest numbers (this is after a reboot).
Code: 
bin/iperf.exe -c 192.174.149.150 -P 1 -i 1 -p 5001 -w 124K -f m -t 20 -T 1
------------------------------------------------------------
Client connecting to 192.174.149.150, TCP port 5001
TCP window size: 0.12 MByte
------------------------------------------------------------
[3960] local 192.174.149.161 port 1126 connected with 192.174.149.150 port 5001
[ ID] Interval       Transfer     Bandwidth
[3960]  0.0- 1.0 sec  62.2 MBytes   522 Mbits/sec
[3960]  1.0- 2.0 sec  66.4 MBytes   557 Mbits/sec
[3960]  2.0- 3.0 sec  64.1 MBytes   538 Mbits/sec
[3960]  3.0- 4.0 sec  71.0 MBytes   596 Mbits/sec
[3960]  4.0- 5.0 sec  70.5 MBytes   591 Mbits/sec
[3960]  5.0- 6.0 sec  67.7 MBytes   568 Mbits/sec
[3960]  6.0- 7.0 sec  69.7 MBytes   584 Mbits/sec
[3960]  7.0- 8.0 sec  66.2 MBytes   555 Mbits/sec
[3960]  8.0- 9.0 sec  72.1 MBytes   605 Mbits/sec
[3960]  9.0-10.0 sec  66.8 MBytes   561 Mbits/sec
[3960] 10.0-11.0 sec  72.3 MBytes   607 Mbits/sec
[3960] 11.0-12.0 sec  70.5 MBytes   592 Mbits/sec
[3960] 12.0-13.0 sec  68.7 MBytes   576 Mbits/sec
[3960] 13.0-14.0 sec  65.5 MBytes   549 Mbits/sec
[3960] 14.0-15.0 sec  64.2 MBytes   539 Mbits/sec
[3960] 15.0-16.0 sec  73.5 MBytes   617 Mbits/sec
[3960] 16.0-17.0 sec  71.4 MBytes   599 Mbits/sec
[3960] 17.0-18.0 sec  70.5 MBytes   591 Mbits/sec
[3960] 18.0-19.0 sec  69.2 MBytes   580 Mbits/sec
[3960] 19.0-20.0 sec  69.2 MBytes   580 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[3960]  0.0-20.0 sec  1372 MBytes   575 Mbits/sec
Done.

LittleMe said:
....
This was just for fun, it's between my 2 servers with 10GbE here at home.
iperf -c 192.168.1.50 -w 2M
------------------------------------------------------------
Client connecting to 192.168.1.50, TCP port 5001
TCP window size: 2.00 MByte
------------------------------------------------------------
[108] local 192.168.1.51 port 54849 connected with 192.168.1.50 port 5001
[ ID] Interval Transfer Bandwidth
[108] 0.0-10.0 sec 11.51 GBytes 9.89 Gbits/sec
Click to expand...
Nice. :)

Overall, it looks like you need good NICs on both ends in order to get good speed.
So, the NIC on my laptop is the culprit here.
 
Overall, it looks like you need good NICs on both ends in order to get good speed.
Click to expand...

I dunno, i have 2 realteks here, 1 on an old asrock board and 1 on a G31 board and even using a cheap netgear 4 port switch i normally get file transfers of 80-85MB's between the two and thats limited by the harddrives i guess.
 
Toytown said:
I dunno, i have 2 realteks here, 1 on an old asrock board and 1 on a G31 board and even using a cheap netgear 4 port switch i normally get file transfers of 80-85MB's between the two and thats limited by the harddrives i guess.
Click to expand...
Well, silly me. :eek:
I just did an internal file transfer on the laptop (copy a file from one folder to another folder), and the best speed I got was ~30MB/sec.

Also, my Broadcom 5754 NICs don't support jumbo frames. :(
This does explain the iPerf scores if one is to follow this chart: http://www.scl.ameslab.gov/Projects/IBMCluster/Benchmarks.html
 
spectrumbx said:
Also, my Broadcom 5754 NICs don't support jumbo frames. :(
This does explain the iPerf scores if one is to follow this chart: http://www.scl.ameslab.gov/Projects/IBMCluster/Benchmarks.html
Click to expand...

Let's see:

ancient document said:
The following graph illustrates the performance of the ethernet interfaces on the cluster. [1] You can expect peak ethernet performance to be about 90Mbps for the fast-ethernet and ~800Mbps for the Gigabit Ethernet. [2] Latency is similar for both at around 110-130 milliseconds. [3] Note that you must use Jumbo Frames (mtu larger than 1500 bytes) to achieve throughput above 400Mbps. [4] However, at 800Mbps (=100MB/sec) the performance of the network is substantially greater than disk IO.
Click to expand...

[1] False. Several sources have already demonstrated > 950 Mb/s.
[2] Not applicable. We're talking about LAN transfers here, not WAN.
[3] False. Several sources demonstrate 900 Mb/s without jumbo frames.
[4] Valid, but not relevant with faster modern drives or RAID configurations.
 
Your laptop is going to be your limiting factor, not your switches nor your server.
 
Madwand said:
Let's see:



[1] False. Several sources have already demonstrated > 950 Mb/s.
[2] Not applicable. We're talking about LAN transfers here, not WAN.
[3] False. Several sources demonstrate 900 Mb/s without jumbo frames.
[4] Valid, but not relevant with faster modern drives or RAID configurations.
Click to expand...

True.
I mean, my own tests showed 575 Mbits/sec without jumbo frames.
However, I think jumbo frames would have helped in my case.

The cpu performance of my laptop directly affected the test results.
I will try to find another system to do the testing with.
 
Wow, the LACP (channel bounding) is working! :)

I setup 3 clients (my laptop and 2 VMs, which are on the server in bridge mode networking) to connect to the server, and I got over 1.3 GBits/sec on the server side. :eek:

So, indeed, the switch and the server are working just fine.
The clients are the limiting factors.
 
Four well-known facts which may help:

1) Some Broadcom NIC's suck.
2) Laptop NIC's are always limited in some way compared to their desktop counterpart, even if Intel branded
*3) A single transfer with IP between two PC's cannot (usually) fill up a gigabit link. Outside of benchmark tests, if you get over 400Mb on a standard "Windows" copy, that is excellent.
4) The benefit of gigabit, on servers, realize their speed on multiple transfers to/from multiple hosts. That is where gigabit is hot.

* Assumes properly tweaked IP stack and NIC drivers. (ie. tuned window size, window scaling, etc)
 
You must log in or register to reply here.
Back
Top