Problem with ZFS write performance

T

thorban

Supreme [H]ardness
Joined
Mar 17, 2000
Messages
4,283
I have a 9 disk raidz using Samsung F4's, FreeBSD 8.1... My read performance is fine, but writing over the network/smb is only about 8MB/s. I have read about the problem with 4k sectors, what I'm wondering is if there is a way to fix this without losing the data on the drives?

zfs_bench.jpg
 
Also, hardware specifications (CPU, RAM, controller the drives are attached to, etc) would be nice. Any tuning you've done for running ZFS on this system?
 
Code: 
Eric@bsdserver[~]
[17:45]:zpool status
  pool: storage
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad10    ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0

errors: No known data errors

CPU is a single core sempron, 8gb of ram, 8 drives on intel sas controller flashed with LSI IT firmware, 1 drive on the motherboard sata. Though, apparently this is due to a memory issue. I checked top last night and it only listed about 50mb free memory. I rebooted the machine and the write performance increased to ~250MB/s. So I'll have to look for a memory leak somewhere.

Any advice on tuning that I should do? I'm happy with the read/write performance, just need to find a way to make it stay at that level instead of degrading.

Code: 
last pid:  4903;  load averages:  0.00,  0.07,  0.07    up 0+23:38:27  17:48:38
51 processes:  1 running, 50 sleeping
CPU:  0.0% user,  0.0% nice,  0.8% system,  1.5% interrupt, 97.7% idle
Mem: 46M Active, 49M Inact, 2189M Wired, 412K Cache, 776M Buf, 5130M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  940 root        4  44    0  5804K  1428K rpcsvc   5:15  0.00% nfsd
 1105 root        1  44    0 11864K  2828K select   0:02  0.00% ntpd
  807 bind        4  44    0 34556K 26124K kqread   0:02  0.00% named
  464 _pflogd     1  44    0  8112K  1728K bpf      0:02  0.00% pflogd
 1200 root        1  44    0 77640K  7284K select   0:01  0.00% httpd
  976 root        1  44    0 23088K  4900K select   0:01  0.00% nmbd
 1259 root        1  44    0 12096K  4196K select   0:01  0.00% sendmail
 1309 root        1  44    0  5936K  1464K nanslp   0:00  0.00% inadyn
  722 root        1  44    0  7024K  1592K select   0:00  0.00% syslogd
 1274 root        1  44    0  7952K  1624K nanslp   0:00  0.00% cron
 1171 dhcpd       1  44    0 10644K  5896K select   0:00  0.00% dhcpd
 1096 _dhcp       1  44    0  5808K  1668K select   0:00  0.00% dhclient
  822 root        1  44    0  7956K  1656K select   0:00  0.00% rpcbind
 4879 Eric        1  44    0 13352K  4072K pause    0:00  0.00% zsh
 1310 root        1  44    0  9008K  1684K select   0:00  0.00% inetd
  938 root        1  76    0  5804K  1520K select   0:00  0.00% nfsd
 4875 root        1  44    0 38104K  5172K sbwait   0:00  0.00% sshd
 
I assume you run amd64 (64-bit)? If not, do not continue with following advice; i386 needs much different tuning.

You could try tuning settings, put in /boot/loader.conf:
vm.kmem_size="9G" # set to 1.5 times your RAM size
vfs.zfs.arc_min="2G"
vfs.zfs.arc_max="4G"

Also, if you use both UFS and ZFS, it would steal some memory away from ZFS. This can grow very large for larger UFS filesystems. In your case it is only 49MB though.

During the dd write running for at least a minute, what is the system load?

Trying the tuning suggestions and rebooting, what speeds do you get?
 
This is top during a dd:

Code: 
last pid:  7503;  load averages:  5.20,  1.44,  0.67    up 1+00:02:26  18:12:37
55 processes:  5 running, 50 sleeping
CPU:  0.0% user,  0.0% nice, 76.8% system,  4.4% interrupt, 18.8% idle
Mem: 52M Active, 77M Inact, 2237M Wired, 524K Cache, 776M Buf, 5047M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 7502 root        1  72    0  9904K  3124K RUN      0:12 16.55% dd
  940 root        4  44    0  5804K  1428K rpcsvc   5:18  0.00% nfsd
 1105 root        1  44    0 11864K  2828K RUN      0:02  0.00% ntpd
  807 bind        4  44    0 34556K 26124K kqread   0:02  0.00% named
  464 _pflogd     1  44    0  8112K  1728K bpf      0:02  0.00% pflogd
 1200 root        1  44    0 77640K  7284K RUN      0:01  0.00% httpd
  976 root        1  44    0 23088K  4900K select   0:01  0.00% nmbd
 1259 root        1  44    0 12096K  4196K select   0:01  0.00% sendmail
 1309 root        1  44    0  5936K  1464K RUN      0:00  0.00% inadyn
  722 root        1  44    0  7024K  1592K select   0:00  0.00% syslogd
 1274 root        1  44    0  7952K  1624K nanslp   0:00  0.00% cron
 1171 dhcpd       1  44    0 10644K  5896K select   0:00  0.00% dhcpd
 1096 _dhcp       1  44    0  5808K  1668K select   0:00  0.00% dhclient
 4879 Eric        1  44    0 13352K  4072K pause    0:00  0.00% zsh
 4878 Eric        1  44    0 38104K  5188K select   0:00  0.00% sshd
  822 root        1  44    0  7956K  1656K select   0:00  0.00% rpcbind
 1310 root        1  44    0  9008K  1684K select   0:00  0.00% inetd

I am running amd64 so I will apply those settings and report back.

* After applying and rebooting my write is at 333MB/s. Looks very good to me.
Code: 
[18:29]:sudo dd if=/dev/zero of=/storage/zerofile.003 bs=2M count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 62.950713 secs (333141898 bytes/sec)
 
Last edited:
Much better. :)

Note that ZFS and FreeBSD really like multiprocessing (SMP), as you can see the load average during the test was 5.00+, meaning that processes on average have to wait 5 clockticks before it's 'their turn' - this is a sign that a CPU with more cores would increase performance. But if you're happy with this kind of perperformance i'd stick to what you got now; it ought to be sufficient. I would avoid heavy CPU stuff though, like the common usage of compression or deduplication.
 
You must log in or register to reply here.
Back
Top