![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
I have 5 different ZFS servers now running on various CPU/motherboard combinations as well as different versions of Solaris, OI and FreeBSD.
I have been having a nightmare of performance problems with the one I have running OI151a4. I thought these was related to the backreved LSI HBA firmware that I was running...I finally got that HBA replaced with an LSI 9200-8e running the latest firmware (v14) and am STILL having problems.
Basically, it seems that the server is sluggish even when there is apparently no load on the server at all and as soon as you put even a very light load of NFS activity (e.g. 1MB/s of writes) performance immediately goes in the toilet.
The behavior seems very similar to what others have reported as a result of bad 'cstates' power management under various flavors of Solaris (I'm heard that Nexenta best practices include disabling c-states in the BIOS).
I've not only disabled cstates on the motherboard (Supermicro X8DLT), but have also disabled power management in /etc/power.conf.
I'm not sure if this is normal or a sympton of the problem, but when I run the hotkernel Dtrace script, it always shows a high rates of hits on 'unix`i86_mwait':
When I run hotkernel on a Solaris 10 box on an idle machine, all the time is spent in cpu_idle, which would seem as expected. Confusingly (even with Cstates disabled int he BIOS, CPU spends most of it's time in C1 state):
Here's Solaris 11 (power management is in the default setup as I'm not even sure how to disable on S11)...it seems to be doing the same as OI151, 'mwait':
Here's FreeBSD9 where we also get some 'mwait' but almost all is in a different entry point 'acpi_cpu_c1' which I presume is indicating cpu is in C1 state. Obviously, this is a different implementation of ZFS so again, not sure if this is expected behavior.
Anyone know if high 'mwait' is normal for idle system?
Can others try running hotkernel on an idle system and post what you see?
Plz indicate your os version, Cstates BIOS settings, and power.conf settings.
You can get Dtrace tool kit here:
http://www.brendangregg.com/DTraceToolkit-0.99.tar.gz
I have been having a nightmare of performance problems with the one I have running OI151a4. I thought these was related to the backreved LSI HBA firmware that I was running...I finally got that HBA replaced with an LSI 9200-8e running the latest firmware (v14) and am STILL having problems.
Basically, it seems that the server is sluggish even when there is apparently no load on the server at all and as soon as you put even a very light load of NFS activity (e.g. 1MB/s of writes) performance immediately goes in the toilet.
The behavior seems very similar to what others have reported as a result of bad 'cstates' power management under various flavors of Solaris (I'm heard that Nexenta best practices include disabling c-states in the BIOS).
I've not only disabled cstates on the motherboard (Supermicro X8DLT), but have also disabled power management in /etc/power.conf.
I'm not sure if this is normal or a sympton of the problem, but when I run the hotkernel Dtrace script, it always shows a high rates of hits on 'unix`i86_mwait':
Code:
DTraceToolkit-0.99# uname -a
SunOS zulu01 5.11 oi_151a4 i86pc i386 i86pc Solaris
...
unix`mutex_enter 14 0.0%
unix`page_nextn 35 0.0%
genunix`fsflush_do_pages 65 0.1%
unix`i86_mwait 87048 99.7%
# tail /etc/power.conf
# 2012-05-29 - try disable power mgmt
cpupm disable
#cpupm enable poll-mode
# 2012-05-29 - disable deep idle as per http://wesunsolve.net/bugid/id/6958068
cpu_deep_idle disable
# dd if=/rz2pool/kvmdiskimages/winxpmalnet/winxp.qcow2 of=/dev/null bs=128k
1248+0 records in
1248+0 records out
163577856 bytes (164 MB) copied, 0.650629 s, 251 MB/sWhen I run hotkernel on a Solaris 10 box on an idle machine, all the time is spent in cpu_idle, which would seem as expected. Confusingly (even with Cstates disabled int he BIOS, CPU spends most of it's time in C1 state):
Code:
# uname -a
SunOS zulu03 5.10 Generic_142910-17 i86pc i386 i86pc
...
unix`mutex_delay_default 104 0.1%
genunix`avl_find 161 0.1%
genunix`avl_destroy_nodes 294 0.2%
unix`cpu_idle 174545 99.1%
# powertop
C-states (idle power) Avg Residency P-states (frequencies)
C0 (cpu running) (0.7%)
C1 2.2ms (99.3%)Here's Solaris 11 (power management is in the default setup as I'm not even sure how to disable on S11)...it seems to be doing the same as OI151, 'mwait':
Code:
# uname -a
SunOS zulu04 5.11 11.0 i86pc i386 i86pc
...
unix`hat_tlb_inval 68 0.2%
unix`mutex_enter 98 0.2%
unix`do_splx 168 0.4%
unix`i86_mwait 41303 97.7%
Cstates are definitely enabled here as evidenced running powertop:
(I have not yet disabled Cstates in the BIOS on this server)
# powertop
C-states (idle power) Avg Residency P-states (frequencies)
C0 (cpu running) (0.0%)
C1 1.0ms (17.5%)
C2 2.3ms (38.8%)
C3 2.7ms (43.7%)Here's FreeBSD9 where we also get some 'mwait' but almost all is in a different entry point 'acpi_cpu_c1' which I presume is indicating cpu is in C1 state. Obviously, this is a different implementation of ZFS so again, not sure if this is expected behavior.
Code:
zulu02# uname -a
FreeBSD zulu02 9.0-RELEASE-p4 FreeBSD 9.0-RELEASE-p4 #2: Sun Sep 9 23:27:59 EDT 2012 root@zulu02:/usr/obj/usr/src/sys/CAPTURE amd64
0xffffffff81a6f40f 897 0.4%
kernel`hpet_get_timecount 2483 1.1%
kernel`cpu_idle_mwait 18401 7.9%
kernel`acpi_cpu_c1 205269 88.6%Anyone know if high 'mwait' is normal for idle system?
Can others try running hotkernel on an idle system and post what you see?
Plz indicate your os version, Cstates BIOS settings, and power.conf settings.
You can get Dtrace tool kit here:
http://www.brendangregg.com/DTraceToolkit-0.99.tar.gz