ZFS writes nearly completely blocked

packetboy

Limp Gawd
Joined
Aug 2, 2009
Messages
288
Any idea what's going on...all of a sudden ZFS server gets in mode where all write activity is severely blocked...see test below..takes 3 seconds to write 32K. Nothing else really running on the box...

Code:
[Sun May 06 05:40:39 root@zulu01:~/DTraceToolkit-0.99 ] # lockstat -kIW -D 20 time dd if=/dev/zero of=/rz2pool/cytel/temporary/ddtest7.dd bs=32000 count=1
1+0 records in
1+0 records out

real        2.9
user        0.0
sys         0.0

Profiling interrupt: 2312 events in 2.987 seconds (774 events/sec)

Count indv cuml rcnt     nsec Hottest CPU+PIL        Caller                  
-------------------------------------------------------------------------------
 2141  93%  93% 0.00     4656 cpu[6]                 cpu_idle                
  140   6%  99% 0.00     2940 cpu[4]                 avl_destroy_nodes       
    8   0%  99% 0.00     2508 cpu[4]                 mutex_enter             
    4   0%  99% 0.00     2793 cpu[4]                 kmem_cache_free         
    4   0%  99% 0.00     2459 cpu[4]                 mutex_exit              
    2   0%  99% 0.00     2438 cpu[4]                 kmem_free               
    1   0%  99% 0.00     5630 cpu[0]                 fsflush_do_pages        
    1   0% 100% 0.00    13029 cpu[2]                 avl_rotation            
    1   0% 100% 0.00    23998 cpu[1]+11              _resume_from_idle       
    1   0% 100% 0.00     5856 cpu[0]                 page_nextn              
    1   0% 100% 0.00     2475 cpu[0]                 atomic_cas_64           
    1   0% 100% 0.00     5649 cpu[1]                 prefetch_page_r         
    1   0% 100% 0.00     4312 cpu[0]                 tsc_read                
    1   0% 100% 0.00      528 cpu[7]+5               ddi_get32               
    1   0% 100% 0.00    96645 cpu[7]+5               sd_return_command       
    1   0% 100% 0.00     3017 cpu[4]                 space_map_vacate        
    1   0% 100% 0.00    21462 cpu[5]                 arc_write_done          
    1   0% 100% 0.00     5658 cpu[2]                 (usermode)              
    1   0% 100% 0.00    17515 cpu[3]                 scsi_init_pkt           
-------------------------------------------------------------------------------

Code:
# .hotkernel

genunix`avl_remove                                         12   0.1%
unix`mutex_enter                                           21   0.2%
genunix`avl_rotation                                       38   0.4%
zfs`space_map_remove                                      110   1.1%
genunix`avl_insert                                        206   2.0%
zfs`space_map_seg_compare                                 232   2.3%
genunix`avl_walk                                          241   2.3%
zfs`metaslab_segsize_compare                              270   2.6%
genunix`avl_find                                          346   3.4%
unix`cpu_idle                                            8683  84.4%
 
It seemed like from the above diag info that the problem might be related to the pool being close to full (there was only 79GB free from at 12TB pool). I deleted 1TB of stuff and now everything seem fine:

Code:
lockstat -kIW -D 20 time dd if=/dev/zero of=/rz2pool/cytel/temporary/ddtest6.dd bs=32000 count=1

1+0 records in
1+0 records out

real        0.0
user        0.0
sys         0.0

Profiling interrupt: 80 events in 0.105 seconds (759 events/sec)

Count indv cuml rcnt     nsec Hottest CPU+PIL        Caller                  
-------------------------------------------------------------------------------
   78  98%  98% 0.00     2440 cpu[6]                 cpu_idle                
    1   1%  99% 0.00     2046 cpu[0]                 page_downgrade          
    1   1% 100% 0.00     1640 cpu[0]                 (usermode)              
-------------------------------------------------------------------------------
 
Back
Top