10GbE Performance = ~800-865MiB/s

W

war9200

Limp Gawd
Joined
Oct 9, 2010
Messages
273
Full thread here, started with 250MiB/s or so:
https://lkml.org/lkml/2011/2/5/159

Two SuperMicro motherboards.


Hi,

1. Defaults below:
sysctl -w net.core.wmem_max=131071
sysctl -w net.core.rmem_max=131071
sysctl -w net.core.wmem_default=118784
sysctl -w net.core.rmem_default=118784
sysctl -w net.core.optmem_max=20480
sysctl -w net.ipv4.igmp_max_memberships=20
sysctl -w net.ipv4.tcp_mem="379104 505472 758208"
sysctl -w net.ipv4.tcp_wmem="4096 16384 4194304"
sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"
sysctl -w net.ipv4.udp_mem="379104 505472 758208"
sysctl -w net.ipv4.udp_rmem_min=4096
sysctl -w net.ipv4.udp_wmem_min=4096
sysctl -w net.core.netdev_max_backlog=1024

2. Optimized settings, for > 800MiB/:

# for 3ware raid, use 16384 readahead, > 16384 readahead, no improvement
blockdev --setra 16384 /dev/sda

# not sure if this helps much
ethtool -K eth0 lro on

# seems to get performance > 600-700MiB/s faster
sysctl -w net.core.wmem_max=4194304
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_default=4194304
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.optmem_max=20480
sysctl -w net.ipv4.igmp_max_memberships=20
sysctl -w net.ipv4.tcp_mem="4194304 4194304 4194304"
sysctl -w net.ipv4.tcp_wmem="4194304 4194304 4194304"
sysctl -w net.ipv4.tcp_rmem="4194304 4194304 4194304"
sysctl -w net.ipv4.udp_mem="4194304 4194304 4194304"
sysctl -w net.ipv4.udp_rmem_min=4096
sysctl -w net.ipv4.udp_wmem_min=4096
sysctl -w net.core.netdev_max_backlog=1048576

# the main option that makes all of the difference, the golden option
# is the rszie and wsize of 1megabyte below:
10.0.1.4:/r1 /nfs/mru02/r1 nfs tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=1048576,wsize=1048576 0 0

CPU utilization:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2069 root 20 0 18640 1304 688 R 91 0.0 0:15.50 cp
703 root 20 0 0 0 0 S 25 0.0 2:46.95 kswapd0

With a single copy I get roughly 700-800MiB/s:

Device eth0 [10.0.1.3] (1/1):
================================================================================
Incoming:
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### ####
###################### #################### #### Curr: 808.71 MByte/s
###################### #################### #### Avg: 706.11 MByte/s
###################### #################### #### Min: 0.00 MByte/s
###################### #################### #### Max: 860.17 MByte/s
###################### #################### #### Ttl: 344.70 GByte

With two copies I get up to 830-850MiB/s:

Device eth0 [10.0.1.3] (1/1):
================================================================================
Incoming:
############################################ ####
############################################ ####
############################################ ####
############################################ ####
############################################ ####
############################################ #### Curr: 846.61 MByte/s
############################################ #### Avg: 683.14 MByte/s
############################################ #### Min: 0.00 MByte/s
############################################ #### Max: 860.17 MByte/s
############################################ #### Ttl: 305.71 GByte

Using a 4MiB r/w size with NFS improves performance to sustain > 750MiB/s
a little better I think:
10.0.1.4:/r1 /nfs/mru02/r1 nfs tcp,bg,rw,hard,intr,nolock,nfsvers=3,rsize=4194304,wsize=4194304 0

Anyhow, roughly 750-850MiB/s it would be nice to get 1Gbyte/sec but I guess
the kerrnel (or my HW, CPU not fast enough) is not there yet.

Justin.
 
So what NIC do you use?

Your link only says "NIC = 10GbE AT2 Server Adapter". Could you give us brand+type? Howmuch did you pay for the card?

I've been looking for the right moment to join 10GbE, but so far i found NICs to be around 600 euro and switches be about 1200 euro, which is too expensive i think. I would like NICs to be at 120 euro and switches at around 200 or so (8-port); though i probably join 10GbE before hitting these numbers.
 
That's the model...Intel AT2. I have a half dozen Intel 10gbit NICs sitting on my desk at the moment that I've been playing with. Guess I should post my results sometime soon. Teaming works too:

10gbeteaming.png
 
sub.mesa said:
:eek:

*drool*
Click to expand...


lol you DO know that guy's fooking nuts, right? Certifiably insane, IMO. The bastard actually has a crapload of 10GbE NIC and wants to run tests just for shits'n gigs and take pictures... NO... I'M NOT JOKING
 
Perhaps he's crazy enough to 'borrow' me one. :D

Either way, i would love to get 10GbE! But since switches are so expensive, how do you connect more than 2 computers in 10GbE environment? You can crosslink ofcourse, but with 3 PCs that would require dual-port adapters already. So we kind of need affordable switches as well.

FYI: i know the fibre/CX4 stuff is quite common, but i'm interested in 10GBaseT ("copper") standard, which should take over from the widely adopted 1000BaseT at some point.
 
sabregen said:
lol you DO know that guy's fooking nuts, right? Certifiably insane, IMO. The bastard actually has a crapload of 10GbE NIC and wants to run tests just for shits'n gigs and take pictures... NO... I'M NOT JOKING
Click to expand...
Just to prove a point, 40gbit works too apparently. :p

40gbeteaming.png


P1000442.jpg
 
LOL... the man buys 10GbE nic and tests them on a single CPU setup.... YA CHEAP BASTARD
 
the single socket Intel shit is probably faster anyways, lol (actually, i KNOW it is).
 
zeroARMY said:
Amazing! How did you combine them? Also do you have a 10gbe switch?
Click to expand...
Pretty much all Intel server NICs support teaming, so you just have to enable it in the software. I sold my last 10gbit switch back in 2008, so I've done all the testing with direct links.
 
Is that wireless 10 Gbps? I don't see any cables going to the network cards.... :D
 
sabregen said:
lol you DO know that guy's fooking nuts, right? Certifiably insane, IMO. The bastard actually has a crapload of 10GbE NIC and wants to run tests just for shits'n gigs and take pictures... NO... I'M NOT JOKING
Click to expand...

hahahah
 
You must log in or register to reply here.
Back
Top