ZFS Write performance issues

[Dr.Evil]

I have a storage box with the following specs:

2 x 240 Opterons
16 GB DDR 400 ECC
2 1068e in IT mode PCIe based
10 750 Sata II disks
Emulex LP10000DC Fibre Channel Card with 2 ports

All of this running with the latest version of NexentaCore configured with Napp-It

When I had all of the disks in two 5 disk RaidZ1 config I received regular dips in performance when writing served over 2GB FC via Comstar

Then I split the 2 RaidZ1 arrays into individuals to try and eliminate the disks as issues but the same issue occurs and it seemed the issue is still there

I have presented the FC disks both as a regular volume LU and a Thin Provisioned LU and both have the same issue.

Here you can see the write graph from HD Tune Pro:

What could be the issue?

Let me know if you need anymore information.

 

[sub.mesa]

Transaction groups, the txg synctime option would be the parameter you can play with to tweak this behavior. Not sure how tweaking works in Solaris; it's different from BSD. Perhaps gea or others can comment on this.

 

[Dr.Evil]

Transaction groups, the txg synctime option would be the parameter you can play with to tweak this behavior. Not sure how tweaking works in Solaris; it's different from BSD. Perhaps gea or others can comment on this.

I think I found what I needed and was able to tweak the zfs_write_limit_override value and this has smoothed things out.

What should I expect out of a pool of 5 Seagate 750 GB ES in a RaidZ1 for throughput for writes and reads?

 

[sub.mesa]

Assuming your disks can do 100MB/s, you could expect something like 350MB/s read 280MB/s write or so.

 

[Dr.Evil]

Assuming your disks can do 100MB/s, you could expect something like 350MB/s read 280MB/s write or so.

Then there must be something else as I am not seeing that over the FC connection and it should be able to provide up to 200 MBs

I am using an older 940 board so it could be the PCIe bandwidth

 

[Dr.Evil]

Assuming your disks can do 100MB/s, you could expect something like 350MB/s read 280MB/s write or so.

Then there must be something else as I am not seeing that over the FC connection and it should be able to provide up to 200 MBs

I am using an older 940 board so it could be the PCIe bandwidth

Think I might have narrowed it down.

I deleted the pool and then presented the disk as raw devices to COMSTAR via FC and iSCSI and the read and writes are horrible. So it may be a lower level issue and will double check the settings on the 1068e cards.

 

[sub.mesa]

You should distinguish local performance from remote (over the network) performance. Local performance could be 350MB/s, but performance over iSCSI would be way lower than that since your network and iSCSI overhead is involved. Over gigabit you should still be able to get decent speeds though (90MB/s read 80MB/s write or so). You should test on your windows client with CrystalDiskMark or AS SSD; that would ensure you test on the filesystem and not do raw I/O over iSCSI which may be quite slow (queue depth=1).

Not sure about iSCSI performance on Solaris platform though.

 

[Dr.Evil]

You should distinguish local performance from remote (over the network) performance. Local performance could be 350MB/s, but performance over iSCSI would be way lower than that since your network and iSCSI overhead is involved. Over gigabit you should still be able to get decent speeds though (90MB/s read 80MB/s write or so). You should test on your windows client with CrystalDiskMark or AS SSD; that would ensure you test on the filesystem and not do raw I/O over iSCSI which may be quite slow (queue depth=1).

Not sure about iSCSI performance on Solaris platform though.

When it was served over my 2GB FC card it showed a bit better performance, +5-10MBs, but not much

 

[sub.mesa]

But how did you test performance? And are the local (server) speeds okay?

Did you try a simple dd on your ZFS pool locally:
dd if=/dev/zero of=/path/to/pool/zerofile.000 bs=1m count=10000

may need to be bs=1M for Solaris, not sure. This command creates a zerofile.000 file and writes 10GB of zeroes to it. Substitute the path to where the pool is mounted. Then you can read the file again with this command:
dd if=/path/to/pool/zerofile.000 of=/dev/null bs=1m

 

[Dr.Evil]

But how did you test performance?

using hd tune

 

[sub.mesa]

HDTune would use Raw I/O. Only the HDTune Pro "File" benchmark would be appropriate.

 

[Dr.Evil]

HDTune would use Raw I/O. Only the HDTune Pro "File" benchmark would be appropriate.

even when I tried the file benchmark i dont recall the read or writes going over 100 MBs with the FC connection

 

[sub.mesa]

Okay but were they alot better than 5MB/s? I don't know about FC performance, but generally the steps are:

1. test local server performance (on the ZFS server) with dd or bonnie or something else, should be twice your network interface speeds, preferably
2. test your network performance with iperf so know network is functioning like it should with 95+% effciency.
3. now test filesystem over iscsi performance using CrystalDiskMark or AS SSD

 

[Dr.Evil]

Here is where I am at now:

10 750 GB Seagate disks in 2 RaidZ1 vdevs of 5 disks each

root@san:~# dd if=/dev/zero of=/tank_750/zerofile.002 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 50.7957 seconds, 206 MB/s
root@san:~# dd if=/tank_750/zerofile.002 of=/dev/null bs=1M
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 14.4076 seconds, 728 MB/s
root@san:~#

So it seems the write is way better and the read seems kinda high but that may be because I attached a SSD as read cache

 

[Flintstone]

Why not try Solaris 11 Express? On my setup this is faster than NexentaCore. If you must have a Linux-like OS, why not go the ESXi + Solaris VM route (assuming you can do VT-D)?

 

[Dr.Evil]

Why not try Solaris 11 Express? On my setup this is faster than NexentaCore. If you must have a Linux-like OS, why not go the ESXi + Solaris VM route (assuming you can do VT-D)?

Should have done that as I had my OpenIndiana disk right there but for some reason I picked the NexentaCore disk instead.

I will try Solaris Express tonight.

Question though, I have 2 disks in a mirrored pool if I export that pool can I then install OpenSolaris and then import that pool?

This pool has data I do not want to lose.

 

[_Gea]

about performance and Solaris* tuning

if you use Linux or Free BSD, it's necessary to do some OS-tuning for best ZFS performance.
Thats not needed with Nexenta, OpenIndiana or Solaris. ZFS there is part of the OS/ Kernel
development. The OS is always optimized for best ZFS performance. All available resources like
CPU or RAM are used for ZFS automatically. Minimal performance differences are not worth
to care about.

If you want know about your options, see
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

see also http://ww1.4hf.de/2011/02/kostenlose-nas-software-leistungstest-–-optimierte-nas.html
They have done some tests with FreeNas and tuned Ubuntu. One of the best was Openindiana untuned.

About your config:
if you need performance, you should build your pool only from mirrors instead of raid-z

About Import
You can always import pools with a lower or eqal ZFS version,
compared to the supported ZFS-release of your target OS.
SE11 supports ZFS up to V 31. OpenIndiana Pools up to V. 28.,
Nexenta Pools up to V.26.

Import of a Nexenta-Pool to OpenIndiana is absolut trouble-free.

Gea