Help with Poor ZFS Performance

Joined
May 22, 2006
Messages
20
I posted this earlier and still haven't been able to figure out what is causing the poor ZFS performance in Solaris 11 11/11 and Solaris 11.1 -- does anyone have any ideas?

I recently upgraded from OpenSolaris SNV134 to Solaris 11.1. I also upgraded to OpenIndiana 151a7. I noticed a huge performance drop on my RAIDZ3 (160GB * 11 drives) pool. Has anyone else run into this? What could be the cause? Is there anything I can do or some place I can look to figure out what is causing the slowdown?

I still have the Boot Environments for the older versions so booted into each version of Solaris and ran dd. I don't think it is space utilization on the pool since there's not much space used on it and there weren't any changes happening to the pool between reboots today.

Code:
WRITE:
OpenSolaris SNV134	211 MB/s
Solaris 11 Express	194 MB/s
OpenIndiana 151a7	215 MB/s
Solaris 11 11/11	182 MB/s
Solaris 11.1		150 MB/s

READ:
OpenSolaris SNV134	470 MB/s
Solaris 11 Express	499 MB/s
OpenIndiana 151a7	417 MB/s
Solaris 11 11/11	177 MB/s
Solaris 11.1		276 MB/s

Here are the commands I ran to get the speed numbers. I ran these commands after rebooting and I hadn't done anything else on the system. Additionally, the GDM (graphical desktop) service was running in all the boot environments except for in Solaris 11.1.

Code:
###################################OpenSolaris SNV134#############################
solaris@opensolaris:~$ uname -a
SunOS opensolaris 5.11 snv_134 i86pc i386 i86pc Solaris

solaris@opensolaris:~$ zfs list
NAME                                          USED  AVAIL  REFER  MOUNTPOINT
tank                                         24.0G  1.12T  14.1G  /tank

solaris@opensolaris:~$ pfexec time /usr/gnu/bin/dd if=/dev/zero of=/tank/test2 bs=1024000 count=20000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 97.0556 s, 211 MB/s

real     1:37.0
user        0.0
sys        14.9
solaris@opensolaris:~$ pfexec time /usr/gnu/bin/dd if=/tank/test2 of=/dev/null bs=1024000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 43.5681 s, 470 MB/s

real       43.5
user        0.0
sys        22.6

###################################Solaris 11 Express#############################
solaris@opensolaris:~$ uname -a
SunOS opensolaris 5.11 snv_151a i86pc i386 i86pc Solaris
solaris@opensolaris:~$ zfs list
NAME                                          USED  AVAIL  REFER  MOUNTPOINT
tank                                         9.95G  1.13T   326K  /tank

solaris@opensolaris:~$ pfexec time /usr/gnu/bin/dd if=/dev/zero of=/tank/test2 bs=1024000 count=20000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 105.609 s, 194 MB/s

real     1:45.6
user        0.0
sys        14.6
solaris@opensolaris:~$ pfexec time /usr/gnu/bin/dd if=/tank/test2 of=/dev/null bs=1024000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 41.0762 s, 499 MB/s

real       41.0
user        0.0
sys        21.2

###################################OpenIndiana 151a7##############################
solaris@opensolaris:~$ uname -a
SunOS opensolaris 5.11 oi_151a7 i86pc i386 i86pc Solaris
solaris@opensolaris:~$ zfs list
NAME                                          USED  AVAIL  REFER  MOUNTPOINT
tank                                         9.95G  1.13T   326K  /tank

solaris@opensolaris:~$ pfexec time /usr/gnu/bin/dd if=/dev/zero of=/tank/test2 bs=1024000 count=20000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 95.2864 s, 215 MB/s

real     1:35.2
user        0.0
sys        13.7
solaris@opensolaris:~$ pfexec time /usr/gnu/bin/dd if=/tank/test2 of=/dev/null bs=1024000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 49.111 s, 417 MB/s

real       49.1
user        0.0
sys        21.5

###################################Solaris 11 11/11###############################
solaris@opensolaris:~$ uname -a
SunOS opensolaris 5.11 11.0 i86pc i386 i86pc Solaris
solaris@opensolaris:~$ zfs list
NAME                                          USED  AVAIL  REFER  MOUNTPOINT
tank                                         9.95G  1.13T   326K  /tank

solaris@opensolaris:~$ sudo time /usr/gnu/bin/dd if=/dev/zero of=/tank/test2 bs=1024000 count=20000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 112.352 s, 182 MB/s

real     1:52.3
user        0.0
sys        15.4
solaris@opensolaris:~$ sudo time /usr/gnu/bin/dd if=/tank/test2 of=/dev/null bs=1024000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 115.891 s, 177 MB/s

real     1:55.8
user        0.0
sys        18.6

###################################Solaris 11.1###################################
solaris@opensolaris:~$ uname -a
SunOS opensolaris 5.11 11.1 i86pc i386 i86pc Solaris
solaris@opensolaris:~$ zfs list
NAME                                          USED  AVAIL  REFER  MOUNTPOINT
tank                                         23.2G  1.12T  13.3G  /tank

solaris@opensolaris:~$ sudo time /usr/gnu/bin/dd if=/dev/zero of=/tank/test2 bs=1024000 count=20000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 136.429 s, 150 MB/s

real     2:16.4
user        0.1
sys        29.7
solaris@opensolaris:~$ sudo time /usr/gnu/bin/dd if=/tank/test2 of=/dev/null bs=1024000
20000+0 records in
20000+0 records out
20480000000 bytes (20 GB) copied, 74.2107 s, 276 MB/s

real     1:14.2
user        0.0
sys        16.5
 
Sorry, I had to go out of town for a few days.

Has anyone else experienced this issue?
Are there any particular logs I can look out or diagnostics I can run to figure out what is causing the ZFS slowness?
 
dd is accurate enough for the test they are doing here.

But the test is 20gigs, and the results will only be correct though, if there is only 6gigs of ram or less in the system. Personally I change my ram to 1 or 2gigs when doing these kinds of benchmarks.
 
I only have 4GBs of RAM in this system.

I ran bonnie++ in Solaris 11.1 and OI 151a7 but I'm not sure how to interpret the results:

SunOS opensolaris 5.11 11.1 i86pc i386 i86pc Solaris
Code:
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris      8G 140725  55 144827  69 90562  30 159750  75 277528  26 327.4   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20242  98 +++++ +++ +++++ +++ 15264  99 +++++ +++ +++++ +++
opensolaris,8G,140725,55,144827,69,90562,30,159750,75,277528,26,327.4,2,16,20242,98,+++++,+++,+++++,+++,15264,99,+++++,+++,+++++,+++

SunOS opensolaris 5.11 oi_151a7 i86pc i386 i86pc Solaris
Code:
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
opensolaris      8G 122032  54 191131  35 133930  35 169329  93 349737  42 602.3   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 26473  99 +++++ +++ +++++ +++ 21941  99 +++++ +++  7491  96
opensolaris,8G,122032,54,191131,35,133930,35,169329,93,349737,42,602.3,2,16,26473,99,+++++,+++,+++++,+++,21941,99,+++++,+++,7491,96
 
Back
Top