ZFS with Dedupe performance problem

[wheelz]

So a while back I posted this: http://hardforum.com/showthread.php?t=1678933 but I had few other priorities for a while and had a new baby so I had to drop the issue for a while. However now I started looking back into it.

I have narrowed the issue further by doing some isolating and testing. If I isolate to a single initiator and just do a straight copy of a large file (>1GB) over iSCSI, the performance is great at first and then about half way through drops to a crawl and is unusable (not just slow). It is definitely an issue with the solaris box. I was able to repro the issue on both VMware and Windows initiators and fully tested the networking side. I then started disabling and enabling features and found that the issue only happens when dedup is enabled. Now I know many people will say disable dedup but that is one of the major reasons I decided to go down the ZFS route.

I am guessing it has something to do with the zfs caching not working properly as nothing on the hardware shows being stressed. I have an 8 core Xeon, 24GB of ram, 5 1TB disk RAIDZ with 1 120GB SSD l2arc and a mirrored SSD pair for ZIL log. The normal system stats show the CPU usage is low, available memory is fine, and the latency response for each of the disks is >5ms for the RAIDZ drives and really low for all the SSDs (of course).

At first I thought it could have been this: https://forums.oracle.com/forums/thread.jspa?messageID=10438896 but as I went through the criteria, it doesn't appear that my arc_no_grow is stuck at 1 (always says 0 when I check it). I am a little confused by the meta data information as arc_summary.pl says one thing:

System Memory:
Physical RAM: 24558 MB
Free Memory : 3443 MB
LotsFree: 383 MB

ZFS Tunables (/etc/system):

ARC Size:
Current Size: 5293 MB (arcsize)
Target Size (Adaptive): 17271 MB (c)
Min Size (Hard Limit): 64 MB (zfs_arc_min)
Max Size (Hard Limit): 23534 MB (zfs_arc_max)

ARC Size Breakdown:
Most Recently Used Cache Size: 100% 17271 MB (p)
Most Frequently Used Cache Size: 0% 0 MB (c-p)

ARC Efficency:
Cache Access Total: 1046077829
Cache Hit Ratio: 57% 596338685 [Defined State for buffer]
Cache Miss Ratio: 42% 449739144 [Undefined State for Buffer]
REAL Hit Ratio: 89% 936839934 [MRU/MFU Hits Only]

Data Demand Efficiency: 74%
Data Prefetch Efficiency: 18%

CACHE HITS BY CACHE LIST:
Anon: --% Counter Rolled.
Most Recently Used: 89% 531715292 (mru) [ Return Customer ]
Most Frequently Used: 67% 405124642 (mfu) [ Frequent Customer ]
Most Recently Used Ghost: 0% 0 (mru_ghost) [ Return Customer Evicted, Now Back ]
Most Frequently Used Ghost: 0% 0 (mfu_ghost) [ Frequent Customer Evicted, Now Back ]
CACHE HITS BY DATA TYPE:
Demand Data: 13% 81868573
Prefetch Data: 0% 2812037
Demand Metadata: 85% 509534792
Prefetch Metadata: 0% 2123283
CACHE MISSES BY DATA TYPE:
Demand Data: 6% 28707848
Prefetch Data: 2% 12367722
Demand Metadata: 64% 289636382
Prefetch Metadata: 26% 119027192
---------------------------------------------

and the other commands that the thread says to run show something else for meta:

echo "::memstat" | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 935415 3653 15%
ZFS File Data 4344795 16971 69%
Anon 106156 414 2%
Exec and libs 3927 15 0%
Page cache 20807 81 0%
Free (cachelist) 16522 64 0%
Free (freelist) 859460 3357 14%

Total 6287082 24558
Physical 6287081 24558

echo "::arc" | mdb -k
hits = 596365672
misses = 449759581
demand_data_hits = 81872880
demand_data_misses = 28708795
demand_metadata_hits = 509557424
demand_metadata_misses = 289643606
prefetch_data_hits = 2812085
prefetch_data_misses = 12367820
prefetch_metadata_hits = 2123283
prefetch_metadata_misses = 119039360
mru_hits = 531731327
mru_ghost_hits = 0
mfu_hits = 405136189
mfu_ghost_hits = 0
deleted = 450601758
mutex_miss = 1122748
hash_elements = 1602762
hash_elements_max = 1616722
hash_collisions = 311552522
hash_chains = 415418
hash_chain_max = 10
p = 17271 MB
c = 17271 MB
c_min = 64 MB
c_max = 23534 MB
size = 5300 MB
buf_size = 108 MB
data_size = 4914 MB
other_size = 231 MB
l2_hits = 367801
l2_misses = 449391063
l2_feeds = 2481692
l2_rw_clash = 0
l2_read_bytes = 21792 MB
l2_write_bytes = 31593 MB
l2_writes_sent = 7170
l2_writes_done = 7170
l2_writes_error = 0
l2_writes_hdr_miss = 0
l2_evict_lock_retry = 0
l2_evict_reading = 0
l2_abort_lowmem = 17
l2_cksum_bad = 0
l2_io_error = 0
l2_hdr_size = 45 MB
memory_throttle_count = 0
meta_used = 386 MB
meta_max = 394 MB
meta_limit = 0 MB
arc_no_grow = 0
arc_tempreserve = 0 MB

However I'm not sure if those values are really the same thing. Is there anything that jumps out at anyone (as I'm not sure I know enough to spot a problem) or have any other ideas? Thanks!

 

[danswartz]

There do not turn out to be a lot of cases where dedup is a win, and it still apparently has some rough edges. Unless there is something in your use case that argues for this in a big way, I don't think you will get a lot of help/success.

 

[patrickdk]

This just doesn't sound good at all, unless you modified the default zvol settings, it's using 8k block sizes, those 8k blocks are getting split up into 2k blocks per disk. Read performance will be horrible.

You didn't post any info about your *test*. How large was the *large file*, how far along did it go, till it started to slow down?

I'm guessing the slowdown is when dedup table filled your ram, (all 6gigs by default). In your config if your disks where full, would require 144gigs for the dedup table.

 

[wheelz]

This just doesn't sound good at all, unless you modified the default zvol settings, it's using 8k block sizes, those 8k blocks are getting split up into 2k blocks per disk. Read performance will be horrible.

You didn't post any info about your *test*. How large was the *large file*, how far along did it go, till it started to slow down?

I'm guessing the slowdown is when dedup table filled your ram, (all 6gigs by default). In your config if your disks where full, would require 144gigs for the dedup table.

Actually I did modify the default block size to align it with vmfs and ntfs (to 64k). Does that still give me horrible read performance? If so, why?

I did mention that it was larger than 1 GB. The "test" I was doing wasn't not a performance benchmark but just a way I found to reproduce the spike in latency I was seeing as I was trying to isolate the cause. It seemed to usually happen around the 500MB mark, but it did vary some when I changed the files. Right now I only have about 800GB of actual data on it.

I'm not opposed to adding more memory and planned on doing so as I added more data but I didn't think I was to that point yet. My motherboard can take up to 192GB. Would it be beneficial to tweak the 25% allowed for the dedupe table? or do I really just need to add more memory?

 

[brutalizer]

Instead of adding memory, maybe you could just decrease the zpool size? Export the zpool, and create a new zpool which is much smaller. Then it would fit into RAM and you would see how working set in RAM, affects dedupe performance?

(I would stay away from ZFS dedupe as of now unless you know what you are doing and have loads of RAM.)

 

[madrebel]

dedupe is slow, not just in zfs. why do you think you want dedupe in the first place?

 

[wheelz]

Instead of adding memory, maybe you could just decrease the zpool size? Export the zpool, and create a new zpool which is much smaller. Then it would fit into RAM and you would see how working set in RAM, affects dedupe performance?

(I would stay away from ZFS dedupe as of now unless you know what you are doing and have loads of RAM.)

Correct me if I'm wrong, but I thought that the size of the dedupe table was based on the data, not the disk size. My understanding is that if I decrease the pool size but keep the same data, it won't change the dedupe table size.

 

[madrebel]

not data size so much as block size with 4K block size being the smallest block but largest dedupe table size.

however 4K blocks are the only way you're ever going to see any real benefit out of dedupe.

 

[brutalizer]

Correct me if I'm wrong, but I thought that the size of the dedupe table was based on the data, not the disk size. My understanding is that if I decrease the pool size but keep the same data, it won't change the dedupe table size.

Yes you are correct. I was a bit fuzzy. I meant that you could create a new small zpool and fill it with data, instead of deleting data from your current zpool to make it small.