Poor Performance From Gigabit Switch

A

amp88

Limp Gawd
Joined
Sep 3, 2007
Messages
133
I have just got a D-Link DGS-1008D 8 port gigabit switch and 2 Intel PRO/1000GT NICs and I'm getting poor performance. To try and diagnose the problem I've moved both the PCs to within a few feet of each other. Between PC1 and the switch there's a 1m length of Cat6 cable. Between PC2 and the switch there's a 3m length of Cat6 cable. I have gone into the advanced properties of each of the NICs and changed "Jumbo Packet" to 16128 bytes (i.e. I've enabled jumbo frames on both NICs). The switch is displaying both connections as 1000Mbps and Windows is identifying both connections as 1.0 Gbps too.

When I copy files from PC1 to PC2 I get roughly 25MB/sec. When I copy files from PC2 to PC1 I get roughly 30MB/sec.

I've used the IPERF tool to benchmark the connection and I'm getting 191 Mbits/sec when I run PC2 as server and PC1 as client and 268 Mbits/sec when I run PC1 as the server and PC2 as the client.

PC1 is running Windows Vista SP1 Home Premium 32 bit. PC2 is running Windows XP Pro SP3.

Does anyone have any ideas as to why I'm getting poor performance? Any help or advice would be appreciated.

edit: Just tried the other available options for frame size, results below:

At 4088 bytes I get about 22-23MB/sec transferring from PC1 to PC2 (erratic speed, not steady) and 34MB/sec transferring from PC2 to PC1 (steady).

At 9014 bytes I get 22MB/sec transferring from PC1 to PC2 (steady) and 33MB/sec transferring from PC2 to PC1 (steady).

edit2: This is the drive I'm writing to/ready from on PC1 (RAID0, 2*74GB Raptors) and this is the drive I'm writing to/reading from on PC2 (250GB SATA).
 
amp88 said:
I have gone into the advanced properties of each of the NICs and changed "Jumbo Packet" to 16128 bytes
Click to expand...

16K is too large. It's surprising that you're getting any communication at all. Try setting them to 9K, and confirm that they're actually working using something like the following:

ping -f -l 8900 destination

Try the following command line options for iperf version 1.7:

server: iperf -s
client: iperf -c server -l 64k -t 15 -i 3 -r

Test with JF enabled and disabled to compare results.

amp88 said:
When I copy files from PC1 to PC2 I get roughly 25MB/sec. When I copy files from PC2 to PC1 I get roughly 30MB/sec.
Click to expand...

Your speeds are actually normal for untuned Windows file transfers over gigabit.
 
Don't use Windows file copying as your benchmark; it relies upon too many other factors (hard drive speeds, etc). You should use a tool like iperf that is written to only stress the network.
 
Madwand said:
Try the following command line options for iperf version 1.7:

server: iperf -s
client: iperf -c server -l 64k -t 15 -i 3 -r

Test with JF enabled and disabled to compare results.
Click to expand...

With 9014 bytes I get the following output:

PC1 as client, PC2 as server:

Code: 
E:\Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.2 -l 64k -t 15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[128] local 192.168.1.3 port 50427 connected with 192.168.1.2 port 500
[ ID] Interval       Transfer     Bandwidth
[128]  0.0- 3.0 sec   189 MBytes   527 Mbits/sec
[128]  3.0- 6.0 sec   182 MBytes   508 Mbits/sec
[128]  6.0- 9.0 sec   182 MBytes   508 Mbits/sec
[128]  9.0-12.0 sec   183 MBytes   511 Mbits/sec
[128] 12.0-15.0 sec   182 MBytes   508 Mbits/sec
[128]  0.0-15.0 sec   916 MBytes   512 Mbits/sec
[140] local 192.168.1.3 port 5001 connected with 192.168.1.2 port 1213
[ ID] Interval       Transfer     Bandwidth
[140]  0.0- 3.0 sec   199 MBytes   558 Mbits/sec
[140]  0.0- 3.0 sec  1.67 Gbits   556 Mbits/sec

PC1 as server, PC2 as client:

Code: 
E:\250SATAData\Firefox Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.3 -l 64k -t
15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1860] local 192.168.1.2 port 1214 connected with 192.168.1.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[1860]  0.0- 3.0 sec   205 MBytes   574 Mbits/sec
[1860]  3.0- 6.0 sec   202 MBytes   564 Mbits/sec
[1860]  6.0- 9.0 sec   203 MBytes   566 Mbits/sec
[1860]  9.0-12.0 sec   203 MBytes   569 Mbits/sec
[1860] 12.0-15.0 sec   204 MBytes   570 Mbits/sec
[1860]  0.0-15.0 sec  1017 MBytes   568 Mbits/sec
[1840] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 50492
[ ID] Interval       Transfer     Bandwidth
[1840]  0.0- 3.0 sec   203 MBytes   568 Mbits/sec
[1840]  3.0- 6.0 sec   204 MBytes   570 Mbits/sec
[1840]  6.0- 9.0 sec   205 MBytes   572 Mbits/sec
[1840]  9.0-12.0 sec   207 MBytes   578 Mbits/sec
[1840]  0.0-15.0 sec  1022 MBytes   572 Mbits/sec

With jumbo frames off:

PC1 as client, PC2 as server:

Code: 
E:\Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.2 -l 64k -t 15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[128] local 192.168.1.3 port 50506 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[128]  0.0- 3.0 sec   184 MBytes   516 Mbits/sec
[128]  3.0- 6.0 sec   176 MBytes   493 Mbits/sec
[128]  6.0- 9.0 sec   177 MBytes   496 Mbits/sec
[128]  9.0-12.0 sec   177 MBytes   496 Mbits/sec
[128] 12.0-15.0 sec   177 MBytes   496 Mbits/sec
[128]  0.0-15.0 sec   893 MBytes   499 Mbits/sec
[140] local 192.168.1.3 port 5001 connected with 192.168.1.2 port 1254
[ ID] Interval       Transfer     Bandwidth
[140]  0.0- 3.0 sec   201 MBytes   562 Mbits/sec
[140]  0.0- 3.0 sec  1.69 Gbits   560 Mbits/sec

PC1 as server, PC2 as client:

Code: 
E:\250SATAData\Firefox Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.3 -l 64k -t
15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1860] local 192.168.1.2 port 1214 connected with 192.168.1.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[1860]  0.0- 3.0 sec   205 MBytes   574 Mbits/sec
[1860]  3.0- 6.0 sec   202 MBytes   564 Mbits/sec
[1860]  6.0- 9.0 sec   203 MBytes   566 Mbits/sec
[1860]  9.0-12.0 sec   203 MBytes   569 Mbits/sec
[1860] 12.0-15.0 sec   204 MBytes   570 Mbits/sec
[1860]  0.0-15.0 sec  1017 MBytes   568 Mbits/sec
[1840] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 50492
[ ID] Interval       Transfer     Bandwidth
[1840]  0.0- 3.0 sec   203 MBytes   568 Mbits/sec
[1840]  3.0- 6.0 sec   204 MBytes   570 Mbits/sec
[1840]  6.0- 9.0 sec   205 MBytes   572 Mbits/sec
[1840]  9.0-12.0 sec   207 MBytes   578 Mbits/sec
[1840]  0.0-15.0 sec  1022 MBytes   572 Mbits/sec

Madwand said:
Your speeds are actually normal for untuned Windows file transfers over gigabit.
Click to expand...

I edited my original post to add in some more details (including hard drive read/write speeds). I don't understand why the network transfers are so much slower than the hard drive speeds.
 
Connect the PC's together via just the cable and bypass the switch completely, then try the same tests and posts results. It could very well be the switch is the performance bottleneck.
 
amp88 said:
I don't understand why the network transfers are so much slower than the hard drive speeds.
Click to expand...

Windows file transfers use complicated mechanisms and are often not the fastest, and you're using XP on one end, which was released well before gigabit became common. Even Vista has been something of a work in progress for high speed transfers, drivers, etc.

Your iperf speeds still look somewhat low. Try version 1.7 as I suggested, and try double-checking the NIC properties, perhaps trying different values for interrupt moderation. (We don't need to see detailed data from all of them BTW.)

Once you're done "tuning" the NIC, try just pushes from Vista to XP. Also try enabling "LargeSystemCache" in the registry (search Microsoft for more details on this).

Alternatively, bypass Windows file transfers altogether using third-party ftp applications such as FileZilla server & client.
 
I've just moved the PCs back to where they normally live, so now I'm connecting PC1 to the switch with 3m of Cat6 cable and PC2 to the switch with 15m of Cat6 cable. The speeds have gone massively downhill. I don't understand why there's such a big drop in performance with this 15 metres.

Toytown said:
Connect the PC's together via just the cable and bypass the switch completely, then try the same tests and posts results. It could very well be the switch is the performance bottleneck.
Click to expand...

If I directly connect the two PCs with 15m of Cat6 cable and a frame size of 9014 bytes I get the following results:

PC1 as client, PC2 as server:

Code: 
E:\Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.2 -l 64k -t 15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[128] local 192.168.1.3 port 49293 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[128]  0.0- 3.0 sec  80.3 MBytes   225 Mbits/sec
[128]  3.0- 6.0 sec  80.3 MBytes   225 Mbits/sec
[128]  6.0- 9.0 sec  80.4 MBytes   225 Mbits/sec
[128]  9.0-12.0 sec  80.4 MBytes   225 Mbits/sec
[128] 12.0-15.0 sec  80.3 MBytes   224 Mbits/sec
[128]  0.0-15.0 sec   402 MBytes   225 Mbits/sec
[140] local 192.168.1.3 port 5001 connected with 192.168.1.2 port 1412
[ ID] Interval       Transfer     Bandwidth
[140]  0.0- 3.0 sec  41.3 MBytes   116 Mbits/sec
[140]  0.0- 3.0 sec   347 Mbits   116 Mbits/sec

PC1 as server, PC2 as client:

Code: 
E:\250SATAData\Firefox Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.3 -l 64k -t
15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1856] local 192.168.1.2 port 1413 connected with 192.168.1.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[1856]  0.0- 3.0 sec  41.5 MBytes   116 Mbits/sec
[1856]  3.0- 6.0 sec  41.5 MBytes   116 Mbits/sec
[1856]  6.0- 9.0 sec  41.6 MBytes   116 Mbits/sec
[1856]  9.0-12.0 sec  41.6 MBytes   116 Mbits/sec
[1856] 12.0-15.0 sec  41.6 MBytes   116 Mbits/sec
[1856]  0.0-15.0 sec   208 MBytes   116 Mbits/sec
[1836] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 49438
[ ID] Interval       Transfer     Bandwidth
[1836]  0.0- 3.0 sec  77.4 MBytes   217 Mbits/sec
[1836]  3.0- 6.0 sec  76.0 MBytes   213 Mbits/sec
[1836]  6.0- 9.0 sec  77.5 MBytes   217 Mbits/sec
[1836]  9.0-12.0 sec  77.6 MBytes   217 Mbits/sec
[1836] 12.0-15.0 sec  77.5 MBytes   217 Mbits/sec
[1836]  0.0-15.0 sec   386 MBytes   216 Mbits/sec

With jumbo frames disabled I get the following results:

PC1 as client, PC2 as server:

Code: 
E:\Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.2 -l 64k -t 15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[128] local 192.168.1.3 port 49449 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[128]  0.0- 3.0 sec  80.9 MBytes   226 Mbits/sec
[128]  3.0- 6.0 sec  81.1 MBytes   227 Mbits/sec
[128]  6.0- 9.0 sec  81.3 MBytes   227 Mbits/sec
[128]  9.0-12.0 sec  81.4 MBytes   228 Mbits/sec
[128] 12.0-15.0 sec  81.3 MBytes   227 Mbits/sec
[128]  0.0-15.0 sec   406 MBytes   227 Mbits/sec
[140] local 192.168.1.3 port 5001 connected with 192.168.1.2 port 1453
[ ID] Interval       Transfer     Bandwidth
[140]  0.0- 3.0 sec  41.5 MBytes   116 Mbits/sec
[140]  0.0- 3.0 sec   348 Mbits   116 Mbits/sec

PC1 as server, PC2 as client:

Code: 
E:\250SATAData\Firefox Downloads\jperf-2.0.0\bin>iperf -c 192.168.1.3 -l 64k -t
15 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1868] local 192.168.1.2 port 1451 connected with 192.168.1.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[1868]  0.0- 3.0 sec  41.7 MBytes   117 Mbits/sec
[1868]  3.0- 6.0 sec  41.5 MBytes   116 Mbits/sec
[1868]  6.0- 9.0 sec  41.5 MBytes   116 Mbits/sec
[1868]  9.0-12.0 sec  41.6 MBytes   116 Mbits/sec
[1868] 12.0-15.0 sec  41.4 MBytes   116 Mbits/sec
[1868]  0.0-15.0 sec   208 MBytes   116 Mbits/sec
[1944] local 192.168.1.2 port 5001 connected with 192.168.1.3 port 49448
[ ID] Interval       Transfer     Bandwidth
[1944]  0.0- 3.0 sec  77.7 MBytes   217 Mbits/sec
[1944]  3.0- 6.0 sec  77.5 MBytes   217 Mbits/sec
[1944]  6.0- 9.0 sec  77.4 MBytes   216 Mbits/sec
[1944]  9.0-12.0 sec  77.5 MBytes   217 Mbits/sec
[1944]  0.0-15.0 sec   388 MBytes   217 Mbits/sec

Madwand said:
Your iperf speeds still look somewhat low. Try version 1.7 as I suggested,
Click to expand...

Version output:

Code: 
E:\250SATAData\Firefox Downloads\jperf-2.0.0\bin>iperf -version
iperf version 1.7.0 (13 Mar 2003) win32 threads

Madwand said:
try double-checking the NIC properties, perhaps trying different values for interrupt moderation. (We don't need to see detailed data from all of them BTW.)
Click to expand...

I don't know much about the NIC's properties, so do you have any suggestions for exactly what I should change and what I should change them to? If I go flailing about without knowing what I'm doing I'm likely to bugger things up more :D

I'm only including the entire output because whenever I'm helping someone to diagnose a problem I like to have all available input then I can filter it down myself.

Madwand said:
Also try enabling "LargeSystemCache" in the registry (search Microsoft for more details on this).
Click to expand...

I enabled LargeSystemCache on the XP machine but it made no perceivable difference.

Madwand said:
Alternatively, bypass Windows file transfers altogether using third-party ftp applications such as FileZilla server & client.
Click to expand...

When I run an FTP server on PC2 and download from it to PC1 I get 14MB/sec. When I run a server on PC1 and download from it to PC2 I get 19MB/sec.

Thanks for the help and suggestions so far guys... :)
 
Well, most of my remarks were from when you were getting ~500 Mb/s. As that's gone down significantly, the first thing to do is to figure out how to get basic network / iperf performance back up.

I'd start with checking connections / cables. Move them closer together if needed to test swaps. Perhaps Intel has some diagnostic tools which can help.
 
It´s a stupid question but did you enable the Jumbo Frames in PC1 and 2 NICs?
 
I think this post was answered already ... the HF lost about a weeks worth of posts in their latest (and ongoing) database problems.

If I remember correctly it was windows vista QoS mechanisms throttling the connection to allow streaming media to get through.
 
On a side note, I typically get any where between 25mb/sec and 50mb/sec over my gigabit Netgear switch between Vista and Server 2008 clients.
 
archivalbackup said:
I think this post was answered already ... the HF lost about a weeks worth of posts in their latest (and ongoing) database problems.

If I remember correctly it was windows vista QoS mechanisms throttling the connection to allow streaming media to get through.
Click to expand...
I'll have to try this when I get home because transferring a 450MB file to my NAS was going extremely slow.
 
MotionBlur said:
On a side note, I typically get any where between 25mb/sec and 50mb/sec over my gigabit Netgear switch between Vista and Server 2008 clients.
Click to expand...

if that is MB/s that's pretty decent. On enterprise class switches, servers and SAN I can get about 90-100MB/s on a single gig link, which is pretty awesome (~78% productive use of the link). At home I can max out around 62MB/s to my NAS.
 
You must log in or register to reply here.
Back
Top