Slow ZFS write performance, confused.

[bleomycin]

I've been running zfs on linux for some time across various types of hardware with great success. I recently upgraded my home NAS with 6x 4TB hitachi coolspin drives in raidz2 configuration running ubuntu 12.04 with zfsonlinux. I was previously using 1TB drives in a raidz2 on the same hardware. When the pool was brand new I did a quick dd write test and I don't remember specifics but it was over 200MB/s. Now with slightly under 5TB being used my samba/nfs/local dd write tests are all tapping out at ~45MB/s. The DD test is pegging 1 core at 100% which I found interesting (with or without lz4 enabled) on my Xeon E5530.

The pool was created with ashift=12, i've tried with and without lz4 compression. The disks are connected to an M1015 flashed to IT mode. The system has 8GB of ram installed and i'm not using any crazy features such as dedup, this is purely file server for a home LAN and all i want out of it is consistent gigabit speeds. I'm not really sure what has gone wrong.

 

[HobartTas]

Greetings

Next time when you benchmark it when its near new then also scrub it when there is at least some data on it to see what speed the scrub process gets because you may be suffering from ZFS fragmentation slowdown as described here and performance can drop by a factor of about 10-40 times initial speeds (see graph), this is a problem with all COW (copy on write) filesystems.

The scrubbing process actually walks the directory tree, you will probably find that the speed it does later scrubs at will also fall if it is significantly fragmented. It may also impact re-silvering to a similar extent for exactly the same reason, as you are probably aware random 4K IOPS are considerably slower than reading sequential (or mostly sequential) data.

If it still scrubs at more or less the original high speed then there is probably some other problem involved which I can't help you with.

That's all I can suggest.

Cheers

 

[danswartz]

He is using about 1/3 of the pool. That should not be causing frag issues.

 

[Shockey]

4 1TB drive previously i take it?

 

[bleomycin]

4 1TB drive previously i take it?

5 1TB drives before. I actually ordered 5 more 4TB drives that should be arriving monday or tuesday, I plan to destroy the current pool and go with an 11 drive raidz3. Maybe whatever weird problem happened will solve itself?

 

[Shockey]

5 1TB drives before. I actually ordered 5 more 4TB drives that should be arriving monday or tuesday, I plan to destroy the current pool and go with an 11 drive raidz3. Maybe whatever weird problem happened will solve itself?

Your just throwing money at a problem without knowing the root cause. I can't say if it solve your problem. you will most likely see a performance increase over what you are getting but i don't think it the right solution.

 

[bleomycin]

Your just throwing money at a problem without knowing the root cause. I can't say if it solve your problem. you will most likely see a performance increase over what you are getting but i don't think it the right solution.

Yeah sorry for the confusion, I didn't mean for it to sound like I was just adding drives hoping to increase performance. That was something I was doing anyways and I had actually ordered the drives before discovering the slowdown. I would love to get to the root cause of the problem before they arrive but if not i'll just chuck them in and see what happens.

 

[brutalizer]

Try to boot from another ZFS based liveCD and import your zpool, and redo the benhcmarks? If it is faster, then it is something weird with your OS. If not, there might be problem with hardware?