• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

10GbE NFS performance optimization help

B

bleomycin

Limp Gawd
Joined
Aug 14, 2010
Messages
242
Hello everyone,

I'm running ZoL on a ubuntu 14.04 kvm guest with m1015's being passed through to the vm. Performance has been great. The problem i'm trying to solve now is NFS performance between the storage VM and additional VM's on the same host. Iperf tests between the storage and guest VM's are ~10Gbit/s with no network tuning at all:
Code: 
Client connecting to 192.168.2.3, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.7 port 55137 connected with 192.168.2.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.9 GBytes  10.2 Gbits/sec
Client NFS mount output:

Code: 
192.168.2.3:/tank/Media /media/raid nfs4 rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.7,minorversion=0,local_lock=none,addr=192.168.2.3 0 0

rsize and wsize are set to maximum, async is enabled in the fstab mount:
Code: 
192.168.2.3:/tank/Media     /media/raid     nfs4    _netdev,async,auto    0       0

DD tests from the NFS client, 4k block size:

Code: 
media: Write test (dd): 425.545 MB/s 3404.361 mbps 19.251 seconds 
media: Read test (dd): 454.420 MB/s 3635.362 mbps 18.027 seconds 
media: Copy test (dd): 208.597 MB/s 1668.778 mbps 39.272 seconds

These same tests run on the storage VM natively exceed 1GB/s read/write.

I'm out of ideas but hoping I've missed something obvious? Any help is greatly appreciated!
 
This is probably not related to the storage performance. Virtualization has some latency implications that native solutions do not have.
What Hypervisor do you use? If it is KVM, may one of the following questions give an answer.
Which type of bridge are you using?
Do you use the virtio netdev?
Do you use vhost?
How is your CPU load during the dd tests?

This is what I see between two Ubuntu 12.04.3 VMs on a KVM host:
Code: 
------------------------------------------------------------
Client connecting to apollo, TCP port 5001
TCP window size: 50.3 KByte (default)
------------------------------------------------------------
[  3] local 10.101.10.161 port 56952 connected with 10.101.10.177 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  23.8 GBytes  20.5 Gbits/sec
Between host and guest I get roughly double of that. This is without jumbo frames. The system uses an Ivy Bridge CPU and is heavily optimized. Sometimes I see more that 1 GB/s transfer rates between a Windows 7 guest and the host (where my ZoL runs).
 
Last edited:
omniscence said:
This is probably not related to the storage performance. Virtualization has some latency implications that native solutions do not have.
What Hypervisor do you use? If it is KVM, may one of the following questions give an answer.
Which type of bridge are you using?
Do you use the virtio netdev?
Do you use vhost?
How is your CPU load during the dd tests?
Click to expand...

Thanks for the reply!

1. Hypervisor is KVM, proxmox is the frontend.

2. Using virtio with a network bridge (vmbr1), vhost=on.

3. CPU load on the storage VM is about 50% across 4 cores, so 4 cores at 50% each according to htop when a client is writing to it via NFS. The ZoL volume being written to has compression=lz4 which may account for some of that.

iperf performance is pretty good so i'm just confused at this point. I suppose I could be CPU/hardware limited somewhere along the line though, this hardware is pretty old. 2x E5530's, 48GB DDR3-1066 ECC. I just ran an iperf test on my other machine here with an identical configuration but on newer hardware (e3-1230v2) and achieved 19Gbit/s.
 
Last edited:
My iperf was also running on a E3-1230v2, which explains the almost equal values. I guess that you cannot get much more out of this setup.
I wanted to play around with multiqueue virtio for a long time now, just did not find the time for it. This could maybe improve the troughput.
 
I would imagine close to the same, as he is benifitting from large frame offloading.

If it wasn't for that, you would be capped around 15gbit on that cpu.
 
You must log in or register to reply here.
Back
Top