ZFS+ESXi - debugging NFS latency

[dsumike]

I currently have two OpenIndiana boxes serving up NFS datastores to a 3x ESXi setup for VMs.

• OI-SAN-00: 2x6-core, 32GB, 10x mirrored vdevs (2x 300gb 10k SAS each)
• OI-SAN-01: 2x4-core, 64GB, 20x mirrored vdevs (2x 300gb 10k SAS each)

I did end up disabling sync for now until I can budget for a faster SLOG, but with sync=disabled I have had up to 60+ VM's running concurrently on the OI-SAN-01, each with an individual web/app/db setup with a 4-8GB database. Performance has been fantastic.

My problem is I recently added in the smaller OI-SAN-00, but am seeing some latency issues with only 30 VM's running. They are similar individual web/app/db setups, so there is some moderately heavy DB activity going on...

I checked read/write latency via ESXi, and I am seeing random spikes for both:

Currently, I am running 30 on each, but notice OI-SAN-01 is averaging < 1ms, where OI-SAN-00 is averaging around 10ms, but with spikes showing 200-300+ ms. Throughout the day I monitored this more and saw 700-1400ms spikes as well.

During these spikes, users reported the systems intermediate felt "slow" (which wasn't very helpful), but I want to know if there's anything I can check or tune to help performance?

 

[bds1904]

could be a physical problem with the disk(s) such as a failing disk or really bad vibrations.

 

[dsumike]

could be a physical problem with the disk(s) such as a failing disk or really bad vibrations.

Thanks for the reply. SMART info shows up as all OK in Napp-It. All drives are between 25-28 deg C, good operating temperature.

I will check for noticeable vibration in the morning, though they are all secured pretty well in the supermicro disk trays. I hadn't noticed anything that would indicated vibrations problems last time I looked at it.

I would certainly be open to any other specific suggestions of things to check.

 

[Rectal Prolapse]

In the meantime, you can have a look at DTrace - there are probes you can enable to monitor NFS on the OI side, and it *might* give you a clue to the problem.

 

[dsumike]

In the meantime, you can have a look at DTrace - there are probes you can enable to monitor NFS on the OI side, and it *might* give you a clue to the problem.

Thanks. Any suggestions of what to look at/for?

 

[pmokonino]

A little late to the discussion here, but I though I would post an answer for future reference.

I had this same problem. I used the disklatency.d script from the Dtrace book to identify intermittent high latencies in the slog devices I was using. Here is the link to the script:

http://www.dtracebook.com/index.php/Disk_IO:disklatency.d

For me, it turned out that the vast majority of the blocks written took < 1ms and roughly 3 out of 7000 took 262ms. This was shown in the latency profile on ESXi, but from a performance basis had essentially no impact on user experience as so few of the blocks were delayed.

In your case, where you disabled sync writes, the disklatency.d script should identify for you if you have a bad hard drive that is about to die, or if your system is throttling writes due to high throughput. In addition to looking at latencies, you can check to see how often throttling is occurring by using this script from Adam Leventhal's article on tuning the zfs write throttle(http://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/):

dtrace -n fbt::dsl_pool_need_dirty_delay:return'{ @[args[1] == 0 ? "no delay" : "delay"] = count(); }'

If it is not a bad hard drive but throughput, the options are to increase zfs_dirty_data_max (either by sacrificing some memory from ARC or adding RAM), increase the number of vdevs to increase your pool's write throughput, or add an additional cache of some sort.