Solaris 11 Spiky CIFS Throughput

[taishan]

Hey guys,

I have a bit of an issue with Solaris 11 11/11 running on a...

Dell Poweredge 2950
2x Xeon 5160 (3.0GHz, Dual-Core)
16 GB RAM (4 x 4GB)
2x Integrated Broadcom Netxtreme II NICs
LSI SAS 9201-16e
Norco DS12-D
2x Intel 330 Series 120GB (mirrored boot)
6x Seagate 3TB ST3000DM001

Pool is configured as triple mirror (2 VDEVs, ashift=12). Only sharing protocol used is CIFS using the built-in kernel CIFS engine.

I would post a screenshot of my transfer graphs, but it appears you cannot attach images here. Basically, one second the box is on the roof (floods the gigabit link, 99% utilization). The very next second, it's at 2%. And then back up, and back down. All the way until the transfer is finished. I see about 25% CPU usage when it's on the roof, and CPU usage drops dramatically with the lulls. I suspect something else is interrupting it somehow, but I don't know how to use dtrace to find it.

I also noticed that reading from the pool is slower than from ARC for some reason. I've tried changing hashes etc, to no avail. Not using compression, dedup, or crypto for any throughput tests. rpool scrubs hit over 1GB/s (fletcher4), and dpool scrubs are hitting 600MB/s (sha256, proc capped here - disks can do upward of 900MB/s).

Any help would be greatly appreciated, and will provide more info upon request.

 

[brutalizer]

Google for more information on
zfs_write_limit_override in
/etc/system tunetables

and see if it can be something to tune?

 

[taishan]

brutalizer,

I should probably clarify that this is happening with reads, not writes. Writes appear to be around 8x-9x% of gigabit throughput, then of course write throttle kicks in and lags it down every 5 secs or so (and the txg commits), then picks back up again. That much I expected.

However, I did not at all expect these huge swings in CIFS throughput on reads, especially data coming directly from ARC. I would think if any data should be able to consistently flood the wire, it would be from ARC.

Sometimes, the spikes aren't quite as garish and roller-coaster. Sometimes it hovers in the 8x%, then drops into the 5x-6x%, then back up, and back down (which still sucks IMO). On my built-in SATA controller at home, setting vdev_max_pending=1 seemed to help quite a bit (3x WD30EZRX in triple mirror), but on this LSI SAS controller, it doesn't seem to make any difference.