• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Slow ZFS Send / Recv

TeeJayHoward

TeeJayHoward

Limpness Supreme
2FA
Joined
Feb 8, 2005
Messages
13,326
I've got two servers with Intel 10GigE cards and a Cat6a cable connecting them. iperf shows that the network is capable of 6.17Gbits/sec. Not very fast, but fast enough for me.
Code: 
root@192.168.0.34:~# iperf -c 192.168.0.33
------------------------------------------------------------
Client connecting to 192.168.0.33, TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.34 port 64732 connected with 192.168.0.33 port 5001
write2 failed: Interrupted system call
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.19 GBytes  6.17 Gbits/sec
From what I can tell, I should be able to perform zfs sends at about 243MB/s:
Code: 
root@192.168.0.34:~# zfs send tank@zfs_test_snapshot | pv -i 1 > /dev/null
  11GB 0:00:46 [ 243MB/s] [                                  <=>         ]

Instead, they're going at about 66MB/s!:
Code: 
root@192.168.0.34:~# zfs send tank@zfs_test_snapshot | /usr/local/bin/mbuffer | ssh 192.168.0.33 zfs receive -F tank
in @  0.0 KiB/s, out @ 70.1 MiB/s, 11.0 GiB total, buffer  19% full
summary: 11.0 GiByte in  2min 49.7sec - average of 66.5 MiB/s

Obviously not acceptable. So, what's going on, and how do I fix it? Any ideas?
 
danswartz said:
Encryption overhead? Or am I missing something?
Click to expand...
I guess SSH could be the limiting factor. I tested it using netcat instead of ssh:
Code: 
root@192.168.0.34:~# time zfs send tank@zfs_test_snapshot | pv -i 1 |nc 192.168.0.33 1337
  11GB 0:01:16 [ 148MB/s] [                                      <=>                    ]

real    1m16.346s
user    0m12.566s
sys     1m25.881s
148MB/s. Better, but still not as good as it can get. What else could be getting in the way?
 
I tried both with napp-it.

edit
Performance wise they were similar but more important i found netcat more stable.
 
Last edited:
omniscence said:
Use mbuffer instead of netcat.
Click to expand...
I also received an IM suggesting that I try mbuffer instead of netcat. Does mbuffer have some feature I'm not aware of that can connect two systems together? Or should I be adding mbuffer to the netcat command?
 
Not exactly the newest post but here is a quick and dirty overview of performance ssh/netcat/mbuffer on send/receive here. mbuffer will be the most efficient of the choices as shown. Are you also sure that each end of the cable is set to the same speed/duplex/mtu etc?
 
mwroobel said:
Not exactly the newest post but here is a quick and dirty overview of performance ssh/netcat/mbuffer on send/receive here. mbuffer will be the most efficient of the choices as shown. Are you also sure that each end of the cable is set to the same speed/duplex/mtu etc?
Click to expand...
Yup. net5 below:

Code: 
root@192.168.0.34:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      igb0
net4              Ethernet             up         1000   full      ixgbe0
net2              Ethernet             up         1000   full      igb2
net1              Ethernet             up         1000   full      igb1
net3              Ethernet             up         1000   full      igb3
net5              Ethernet             up         10000  full      ixgbe1
root@192.168.0.34:~# ssh 192.168.0.33 dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      igb0
net4              Ethernet             up         1000   full      ixgbe0
net2              Ethernet             up         1000   full      igb2
net1              Ethernet             up         1000   full      igb1
net3              Ethernet             up         1000   full      igb3
net5              Ethernet             up         10000  full      ixgbe1
root@192.168.0.34:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      9192   up       --
net4                phys      1500   up       --
net3                phys      9192   up       --
net0                phys      9192   up       --
net2                phys      9192   up       --
net5                phys      9192   up       --
aggr0               aggr      9192   up       net0 net1 net2 net3
root@192.168.0.34:~# ssh 192.168.0.33 dladm show-link
LINK                CLASS     MTU    STATE    OVER
net0                phys      9192   up       --
net4                phys      1500   up       --
net2                phys      9192   up       --
net1                phys      9192   up       --
net3                phys      9192   up       --
net5                phys      9192   up       --
aggr0               aggr      9192   up       net0 net1 net2 net3
 
mwroobel said:
Not exactly the newest post but here is a quick and dirty overview of performance ssh/netcat/mbuffer on send/receive here. mbuffer will be the most efficient of the choices as shown.
Click to expand...
Well, hell. Using a modified command from that link looks to have solved my speed issue. Now it's just a matter of finding a way to issue all the commands from one server via cron... Great link. Thanks!

Code: 
root@192.168.0.33:~# /usr/local/bin/mbuffer -I 1337|zfs receive -F tank
in @  266 MiB/s, out @  309 MiB/s, 11.0 GiB total, buffer   0% full
summary: 11.0 GiByte in 43.4sec - average of  260 MiB/s

Code: 
root@192.168.0.34:~# zfs send tank@zfs_test_snapshot|/usr/local/bin/mbuffer -O 192.168.0.33:1337
in @  180 MiB/s, out @  180 MiB/s, 11.0 GiB total, buffer   0% full
summary: 11.0 GiByte in 43.7sec - average of  258 MiB/s


And, it looks like this will work, unless someone knows a better way? There's got to be one...

Code: 
root@192.168.0.33:~# /usr/local/bin/mbuffer -I 1337|zfs receive -F tank & ssh 192.168.0.34 "zfs send tank@zfs_test_snapshot|/usr/local/bin/mbuffer -O 192.168.0.33:1337"
[1] 23763
in @  222 MiB/s, out @  222 MiB/s, 11.0 GiB total, buffer   0% full
summary: 11.0 GiByte in 39.2sec - average of  288 MiB/s

summary: 11.0 GiByte in 39.4sec - average of  287 MiB/s
 
Last edited:
Mbuffer is much faster than netcat, because netcat does not buffer, at least not with a significant size.
This may not matter with simple streams like those produced/consumed with dd, but for zfs send/recv this can be significant.
I also use mbuffer locally for send/recv, because they behave like they do not use a buffer themselves and stall/block each other.

What is wrong with the last command? It looks almost perfect. There is not an elegant way to securely run commands remotely other than that.
You just have to be aware of the fact that the data stream is not encrypted.
 
Last edited:
You must log in or register to reply here.
Back
Top