Any idea what's going on...all of a sudden ZFS server gets in mode where all write activity is severely blocked...see test below..takes 3 seconds to write 32K. Nothing else really running on the box...
Code:
[Sun May 06 05:40:39 root@zulu01:~/DTraceToolkit-0.99 ] # lockstat -kIW -D 20 time dd if=/dev/zero of=/rz2pool/cytel/temporary/ddtest7.dd bs=32000 count=1
1+0 records in
1+0 records out
real 2.9
user 0.0
sys 0.0
Profiling interrupt: 2312 events in 2.987 seconds (774 events/sec)
Count indv cuml rcnt nsec Hottest CPU+PIL Caller
-------------------------------------------------------------------------------
2141 93% 93% 0.00 4656 cpu[6] cpu_idle
140 6% 99% 0.00 2940 cpu[4] avl_destroy_nodes
8 0% 99% 0.00 2508 cpu[4] mutex_enter
4 0% 99% 0.00 2793 cpu[4] kmem_cache_free
4 0% 99% 0.00 2459 cpu[4] mutex_exit
2 0% 99% 0.00 2438 cpu[4] kmem_free
1 0% 99% 0.00 5630 cpu[0] fsflush_do_pages
1 0% 100% 0.00 13029 cpu[2] avl_rotation
1 0% 100% 0.00 23998 cpu[1]+11 _resume_from_idle
1 0% 100% 0.00 5856 cpu[0] page_nextn
1 0% 100% 0.00 2475 cpu[0] atomic_cas_64
1 0% 100% 0.00 5649 cpu[1] prefetch_page_r
1 0% 100% 0.00 4312 cpu[0] tsc_read
1 0% 100% 0.00 528 cpu[7]+5 ddi_get32
1 0% 100% 0.00 96645 cpu[7]+5 sd_return_command
1 0% 100% 0.00 3017 cpu[4] space_map_vacate
1 0% 100% 0.00 21462 cpu[5] arc_write_done
1 0% 100% 0.00 5658 cpu[2] (usermode)
1 0% 100% 0.00 17515 cpu[3] scsi_init_pkt
-------------------------------------------------------------------------------
Code:
# .hotkernel
genunix`avl_remove 12 0.1%
unix`mutex_enter 21 0.2%
genunix`avl_rotation 38 0.4%
zfs`space_map_remove 110 1.1%
genunix`avl_insert 206 2.0%
zfs`space_map_seg_compare 232 2.3%
genunix`avl_walk 241 2.3%
zfs`metaslab_segsize_compare 270 2.6%
genunix`avl_find 346 3.4%
unix`cpu_idle 8683 84.4%