• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Slow ZFS / NFS performance

TeeJayHoward

TeeJayHoward

Limpness Supreme
2FA
Joined
Feb 8, 2005
Messages
13,322
I've got 24x Seagate 2TB SAS drives (Model ST32000444SS) in the ZFS equivalent of a RAID 6+0 under Solaris 11.1 in a ESXi VM. They're in a SuperMicro SC846 chassis, connected to an X10SL7-F motherboard via a fanout cable. Bonnie++ shows that this array is capable of 555 MB/s sequential writes and 834 MB/s sequential reads. More than enough for a gigabit network. However, even with sync=disabled, I'm not getting more than about 25MB/s during an rsync (primarily 200-300MB files). I'd love to know why. Any advice?

Code: 
root@backup:~# zpool status backup
  pool: backup
 state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
        pool will no longer be accessible on older software versions.
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        backup                     ONLINE       0     0     0
          raidz2-0                 ONLINE       0     0     0
            c0t5000C50034F447C7d0  ONLINE       0     0     0
            c0t5000C50034D58277d0  ONLINE       0     0     0
            c0t5000C50034E7A243d0  ONLINE       0     0     0
            c0t5000C50034F37E2Fd0  ONLINE       0     0     0
            c0t5000C50034E7B3AFd0  ONLINE       0     0     0
            c0t5000C50034F435C7d0  ONLINE       0     0     0
            c0t5000C50034FF4993d0  ONLINE       0     0     0
            c0t5000C50034F41723d0  ONLINE       0     0     0
            c0t5000C50034FF2B87d0  ONLINE       0     0     0
            c0t5000C50034F76637d0  ONLINE       0     0     0
            c0t5000C50034E9A92Bd0  ONLINE       0     0     0
            c0t5000C50034FBC2D7d0  ONLINE       0     0     0
          raidz2-1                 ONLINE       0     0     0
            c0t5000C50034EDB303d0  ONLINE       0     0     0
            c0t5000C50034EC07EBd0  ONLINE       0     0     0
            c0t5000C50034F3718Bd0  ONLINE       0     0     0
            c0t5000C50034F4426Bd0  ONLINE       0     0     0
            c0t5000C50034F38ADFd0  ONLINE       0     0     0
            c0t5000C50034FF5EB3d0  ONLINE       0     0     0
            c0t5000C50034F4467Bd0  ONLINE       0     0     0
            c0t5000C50034EB7D9Fd0  ONLINE       0     0     0
            c0t5000C50034F3DD6Bd0  ONLINE       0     0     0
            c0t5000C50034F3DF0Bd0  ONLINE       0     0     0
            c0t5000C50034F43AEBd0  ONLINE       0     0     0
            c0t5000C50034F39007d0  ONLINE       0     0     0

errors: No known data errors
root@backup:~# nfsstat -rc

Client rpc:
Connection oriented:
calls      badcalls   badxids    timeouts   newcreds   badverfs   timers
125460052  2          2          0          0          0          0
cantconn   nomem      interrupts
0          0          2
Connectionless:
calls      badcalls   retrans    badxids    timeouts   newcreds   badverfs
0          0          0          0          0          0          0
timers     nomem      cantsend
0          0          0

root@backup:~# zfs get all backup
NAME    PROPERTY              VALUE                  SOURCE
backup  aclinherit            restricted             default
backup  aclmode               discard                default
backup  atime                 on                     default
backup  available             32.6T                  -
backup  canmount              on                     default
backup  casesensitivity       mixed                  -
backup  checksum              on                     default
backup  compression           off                    default
backup  compressratio         1.00x                  -
backup  copies                1                      default
backup  creation              Fri Sep  5 12:21 2014  -
backup  dedup                 off                    default
backup  devices               on                     default
backup  encryption            off                    -
backup  exec                  on                     default
backup  keychangedate         -                      default
backup  keysource             none                   default
backup  keystatus             none                   -
backup  logbias               latency                default
backup  mlslabel              none                   -
backup  mounted               yes                    -
backup  mountpoint            /backup                default
backup  multilevel            off                    -
backup  nbmand                off                    default
backup  normalization         none                   -
backup  primarycache          all                    default
backup  quota                 none                   default
backup  readonly              off                    default
backup  recordsize            128K                   default
backup  referenced            2.89T                  -
backup  refquota              none                   default
backup  refreservation        none                   default
backup  rekeydate             -                      default
backup  reservation           none                   default
backup  rstchown              on                     default
backup  secondarycache        all                    default
backup  setuid                on                     default
backup  shadow                none                   -
backup  sharenfs              off                    default
backup  sharesmb              off                    default
backup  snapdir               hidden                 default
backup  sync                  disabled               local
backup  type                  filesystem             -
backup  used                  2.89T                  -
backup  usedbychildren        14.2M                  -
backup  usedbydataset         2.89T                  -
backup  usedbyrefreservation  0                      -
backup  usedbysnapshots       354M                   -
backup  utf8only              off                    -
backup  version               5                      -
backup  vscan                 off                    default
backup  xattr                 on                     default
backup  zoned                 off                    default
root@backup:~# zpool iostat backup 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
backup      3.50T  40.0T     14    208   536K  16.3M
backup      3.50T  40.0T      5    495  7.00K  39.8M
backup      3.50T  40.0T      5    427  4.50K  32.0M
backup      3.50T  40.0T      0      0      0      0
backup      3.50T  40.0T      5    340  7.50K  31.8M
backup      3.50T  40.0T      0      0      0      0
backup      3.50T  40.0T      8    621  9.49K  34.0M
backup      3.50T  40.0T      0      0      0      0
backup      3.50T  40.0T      6    375  41.5K  32.9M
backup      3.50T  40.0T      0    166      0  1.17M
root@backup:~# cat /backup/Bonnie.log
Version 1.03e       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
2014.09.07      16G 194584  44 555004  69 385313  41 301472  80 834162  32 547.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  8614  88 +++++ +++  7204  29 29918  86 +++++ +++ +++++ +++
2014.09.07,16G,194584,44,555004,69,385313,41,301472,80,834162,32,547.8,1,16,8614,88,+++++,+++,7204,29,29918,86,+++++,+++,+++++,+++
 
Last edited:
I read something online stating that the Seagate SAS drives had a feature where they would slow down if they detected they were overheating. I made a quick-n-dirty to grab the current drive temp from SMART for each disk. Doesnt look like that's the issue here:

Code: 
root@backup:~# sh drivetemp.sh
Current Drive Temperature:     39 C
Current Drive Temperature:     38 C
Current Drive Temperature:     38 C
Current Drive Temperature:     37 C
Current Drive Temperature:     35 C
Current Drive Temperature:     32 C
Current Drive Temperature:     42 C
Current Drive Temperature:     41 C
Current Drive Temperature:     41 C
Current Drive Temperature:     39 C
Current Drive Temperature:     37 C
Current Drive Temperature:     34 C
Current Drive Temperature:     41 C
Current Drive Temperature:     41 C
Current Drive Temperature:     41 C
Current Drive Temperature:     40 C
Current Drive Temperature:     38 C
Current Drive Temperature:     35 C
Current Drive Temperature:     39 C
Current Drive Temperature:     39 C
Current Drive Temperature:     38 C
Current Drive Temperature:     38 C
Current Drive Temperature:     37 C
Current Drive Temperature:     34 C
 
what kind of speed do you get if you do a large sequential write? that might narrow down the problem area...
 
danswartz said:
what kind of speed do you get if you do a large sequential write?
Click to expand...

Over the network? I created a 10 gig file on one box using dd, and then copied it to the other.
root@nas:~# time dd if=/dev/zero of=/pool/10G.file bs=1024 count=10485760
10485760+0 records in
10485760+0 records out

real 0m50.287s
user 0m2.509s
sys 0m47.537s
Code: 
root@backup:/backup# time cp /net/192.168.0.11/pool/10G.file ./

real    10m17.939s
user    0m0.006s
sys     0m23.692s

16.57MB/s. Horrible.

Locally? I built a 100 gig file. The result:
Code: 
root@backup:/backup# time dd if=/dev/zero of=/backup/100G.file bs=1024 count=104857600
104857600+0 records in
104857600+0 records out

real    8m50.990s
user    0m23.659s
sys     7m38.503s

192.84MB/s. Still not what I'd call "good", but a heck of a lot better than what I was getting over the network.
 
Last edited:
so it's a slow read, not write. have you run iperf to exonerate the network? while running the cp, run iostat to see what the disks are doing?
 
danswartz said:
so it's a slow read, not write. have you run iperf to exonerate the network? while running the cp, run iostat to see what the disks are doing?
Click to expand...
I forgot about iostat... I'll do that in a bit. I filled up my (admittedly small) root partition. I didn't even have enough room to install VMware tools. I decided to try reinstalling from scratch - It doesn't take too long. If it doesn't fix the issue, I'll post up some iostat numbers.
 
Last edited:
just a shot in the dark, but what about alignment is ashift at 9 or 12?
 
Code: 
root@backup:~# zpool import backup
root@backup:~# time cp /net/192.168.0.11/pool/10G.file /backup/

real    1m45.608s
user    0m0.005s
sys     0m8.426s

96.96MB/s. Much, much better!

I ended up installing Solaris 11.2, since I couldn't find my 11.1 .ISO. I then spent a goodly amount of time trying to figure out why the VM wouldn't boot if the controller was passed through. I had forgotten that I copied over the OmniOS bits with my previous install. Thanks to Abisen.com for reminding me.

So, I don't know why it stopped working at a decent speed, but it appears to have resolved itself. I am satisfied.

edit: [size=+2]NOPE, ITS DOING IT AGAIN...[/size]
 
Last edited:
iperf looks good:

Code: 
root@backup:~# iperf -c 192.168.0.11
------------------------------------------------------------
Client connecting to 192.168.0.11, TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.22 port 51514 connected with 192.168.0.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes    933 Mbits/sec
root@backup:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:   125 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.22 port 5001 connected with 192.168.0.11 port 49821
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.09 GBytes    937 Mbits/sec
^Croot@backup:~#
 
oh wait, now i see. this is kind of confusing because you are reading from some other file on an nfs server and writing to the local zfs pool?
 
danswartz said:
oh wait, now i see. this is kind of confusing because you are reading from some other file on an nfs server and writing to the local zfs pool?
Click to expand...
Right. I've got two servers. One is my NAS, the other is my backup server VM. The NAS has all my data on it. The backup server needs all my data. Copying files from one server to the other is not going very well.
 
Shockey said:
Whats your NFS server and client settings?
Click to expand...
Code: 
root@nas:~# zfs get share pool;zfs get sharenfs pool
NAME  PROPERTY  VALUE  SOURCE
pool  share     name=pool,path=/pool,prot=nfs,prot=smb,guestok=true  local
NAME  PROPERTY  VALUE  SOURCE
pool  sharenfs  on     local
No settings on the client side. It's whatever Solaris defaults to. I tried mounting the share and using the /net-directory-automount-thing. No difference.

iostat example: (-xtcM 1)
Code: 
                 extended device statistics                    tty         cpu
device    r/s    w/s   Mr/s   Mw/s wait actv  svc_t  %w  %b  tin tout  us sy st id
sd0       4.0   79.0    0.0    0.1  0.0  0.0    0.1   0   0    0 3720   0  4  0 96
sd1       0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd2       2.0   31.0    0.0    0.1  0.0  0.0    0.8   0   1
sd3       1.0   34.0    0.0    0.1  0.0  0.0    0.9   0   2
sd4       8.0   38.0    0.0    0.5  0.0  0.1    2.4   0   5
sd5       7.0   34.0    0.0    0.5  0.0  0.2    3.8   0   5
sd6       5.0   39.0    0.0    0.5  0.0  0.2    5.7   0   5
sd7       1.0   37.0    0.0    0.1  0.0  0.0    0.6   0   1
sd8       4.0   38.0    0.0    0.5  0.0  0.2    3.8   0   4
sd9       2.0   34.0    0.0    0.5  0.0  0.1    3.9   0   4
sd10      5.0   37.0    0.0    0.5  0.0  0.2    3.8   0   6
sd11      9.0   34.0    0.0    0.5  0.0  0.3    7.3   0  13
sd12      6.0   39.0    0.0    0.5  0.0  0.0    1.1   0   4
sd13      4.0   37.0    0.0    0.5  0.0  0.2    3.9   0   4
sd14      2.0   34.0    0.0    0.1  0.0  0.0    0.5   0   1
sd15      1.0   39.0    0.0    0.1  0.0  0.0    0.7   0   1
sd16      0.0   37.0    0.0    0.1  0.0  0.0    0.5   0   1
sd17      0.0   42.0    0.0    0.1  0.0  0.0    0.6   0   1
sd18      1.0   37.0    0.0    0.1  0.0  0.0    1.2   0   1
sd19      0.0   37.0    0.0    0.1  0.0  0.0    0.8   0   1
sd20      1.0   28.0    0.0    0.1  0.0  0.0    1.4   0   1
sd21      4.0   29.0    0.0    0.5  0.0  0.1    2.4   0   4
sd22      1.0   34.0    0.0    0.1  0.0  0.0    0.6   0   1
sd23      2.0   36.0    0.0    0.1  0.0  0.0    0.5   0   1
sd24      9.0   35.0    0.0    0.5  0.0  0.1    2.0   0   6
sd25      5.0   38.0    0.0    0.5  0.0  0.2    4.5   0  10
nfs4   1151.1    0.0    4.5    0.0  0.0  0.4    0.4   0  25
 
What if you dd from the nfs server to /dev/null (just to eliminate local storage from picture...)
 
danswartz said:
What if you dd from the nfs server to /dev/null (just to eliminate local storage from picture...)
Click to expand...
With an rsync going in the background, even:

Code: 
root@nas:~# time dd if=/pool/10G.file of=/dev/null
20971520+0 records in
20971520+0 records out

real    1m7.862s
user    0m25.417s
sys     0m34.451s

The NFS server can read and write files at over 90MB/s from a (physical) windows client. Whatever's going on only appears to be happening with the Solaris 11.2 VM.
 
And now it's working again. The same rsync that was going so slow last night is now travelling at a decent speed this morning!

Code: 
root@backup:~# zpool iostat backup 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
backup      8.08T  35.4T      4    687  6.84K  79.7M
backup      8.08T  35.4T      0  2.44K      0   306M
backup      8.08T  35.4T      0    767      0  71.0M
backup      8.08T  35.4T      0      0      0      0
backup      8.08T  35.4T      0      0      0      0
backup      8.08T  35.4T      0      0      0      0
backup      8.08T  35.4T      0  2.30K      0   289M
backup      8.09T  35.4T      0  1.04K      0   106M
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0  2.97K      0   374M
backup      8.09T  35.4T      0    491      0  37.0M
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0  2.57K      0   323M
backup      8.09T  35.4T      0  1.05K      0  91.8M
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0      0      0      0
backup      8.09T  35.4T      0      0      0      0
root@backup:~# iostat -xtcM 1
                 extended device statistics                    tty         cpu
device    r/s    w/s   Mr/s   Mw/s wait actv  svc_t  %w  %b  tin tout  us sy st id
sd0       0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0    0 3879  22 23  0 54
sd1       0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
sd2       0.0  140.0    0.0    8.4  0.0  0.5    3.6   0  12
sd3       0.0   81.0    0.0    7.4  0.0  0.5    6.0   0  13
sd4       0.0  187.9    0.0    8.0  0.0  0.5    2.5   0  13
sd5       0.0  148.0    0.0    8.7  0.0  0.4    2.9   0  12
sd6       0.0   86.0    0.0    6.3  0.0  0.5    6.0   0  12
sd7       0.0   82.0    0.0    7.4  0.0  0.5    6.2   0  12
sd8       0.0   71.0    0.0    7.2  0.0  0.5    7.0   0  12
sd9       0.0  180.9    0.0    8.7  0.0  0.4    2.4   0  12
sd10      0.0   77.0    0.0    5.8  0.0  0.6    7.3   0  12
sd11      0.0   62.0    0.0    6.3  0.0  0.5    8.7   0  12
sd12      0.0   60.0    0.0    6.3  0.0  0.6   10.1   0  13
sd13      0.0  157.0    0.0    8.7  0.0  0.5    3.1   0  12
sd14      0.0   96.0    0.0    7.4  0.0  0.5    5.6   0  12
sd15      0.0  185.9    0.0    9.2  0.0  0.5    2.6   0  12
sd16      0.0  189.9    0.0    9.2  0.0  0.5    2.4   0  12
sd17      0.0   99.0    0.0    7.9  0.0  0.5    5.0   0  12
sd18      0.0  176.0    0.0    9.1  0.0  0.5    2.6   0  12
sd19      0.0  142.0    0.0    9.1  0.0  0.5    3.6   0  12
sd20      0.0  152.0    0.0    7.4  0.0  0.5    3.1   0  12
sd21      0.0  129.0    0.0    9.0  0.0  0.6    4.4   0  12
sd22      0.0  317.9    0.0    9.1  0.0  0.3    1.0   0   9
sd23      0.0  110.0    0.0    9.2  0.0  0.6    5.0   0  12
sd24      0.0  170.0    0.0    8.9  0.0  0.5    3.1   0  12
sd25      0.0   68.0    0.0    9.0  0.0  0.6    9.0   0  13
nfs1      0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
nfs2   2693.3    0.0   84.2    0.0  0.0  3.1    1.2   4  97

What the hell?
 
Last edited:
Update: I removed the "VM" aspect entirely. Both machines are now running identical hardware (excepting some unused PCIe cards in one of them) and software. There's a known-good CAT7 cable connecting the two machines. No switch in between.

With a fresh install of Solaris 11.2 on both machines, I can rsync between the two pools at a ridiculously slow speed. I still can't figure out why it's going so slow!
Code: 
root@nas:~# mount 10.0.0.3:/backup /mnt
root@nas:~# rsync -av --delete /mnt/ /pool/
sending incremental file list (<- hangs here)

Code: 
root@nas:~# zpool iostat pool 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
pool        1.43T  42.1T     84    116   192K   465K
pool        1.43T  42.1T    127    716  64.4K  2.50M
pool        1.43T  42.1T    131      0  74.3K      0
pool        1.43T  42.1T     51      0  35.4K      0
pool        1.43T  42.1T     18      0  13.5K      0
pool        1.43T  42.1T     97      0  54.9K      0
^C
Code: 
root@backup:~# zpool iostat backup 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
backup      15.7T  27.8T    190     34  21.5M   135K
backup      15.7T  27.8T      0      0      0      0
backup      15.7T  27.8T    153      0  76.8K      0
backup      15.7T  27.8T    120    725   141K  3.53M
backup      15.7T  27.8T     61      0   228K      0
backup      15.7T  27.8T    115      0   202K      0
^C
 
Looks like I just didn't wait long enough for the initial file list to be transferred. It's going at about 75MB/s.
 
You must log in or register to reply here.
Back
Top