ZFS High CPU Load

[noxxsan]

Hey

Getting issues with this setup:

Solaris - x86
Intel Xeon E31220 @ 3.10GHZ
Motherboard Intel S1200BT
Memory 4x4GB Kingston ECC
LSI SAS 9211-8i HBA x2 Firmware IT Mode
Seagate ST31500341AS 1.50TB x16
Corsair SSD 120GB
Corsair SSD 64GB

Pool 4x1.50 TB Mirror with 2x120GB(mirror) write cache and 64GB SSD read cache.

This what happens when i try to install some VMs.

Due high cpu load packet loss/high network latency and eventually OS installer on VM crash.

Any ideas?


Thanks.

 

[omniscence]

You do not provide enough information. You talk about VMs, but what are the VMs running on and how are they connected to your Solaris server? At the moment this does not look like a ZFS problem.
I encountered a similar problem with ping response times although on a X9SCM-iiF (it seems also on other boards while using the integrated 82574L controller) running Linux. My first suggestion without knowing anything else is to try a dedicated network controller. I switched to an I350 controller till a bugfix is found.

EDIT: You could also try to use the 82579 controller instead (I assume you use the 82574L).

 

[noxxsan]

Two Hyper-V 2008 R2 Servers in cluster mode connected to ZFS via iSCSI

VMs are Windows 2008 R2

Tested both onboard and dedicated network controllers with same issues.

 

[Billy_nnn]

Don't know which version of Solaris you are running, but if you have high CPU use at idle, there were some issues around CPU power management - you can check this by disabling power management and check if the CPU usage falls to very low/near zero at idle.

 

[noxxsan]

ok, reinstalled everything solaris 11.1 and still getting same issues

i dont know why kernel is using up to 10gb memory

 

[paret0]

...

 

[packetboy]

Run the 'hotkernel' Dtrace script from the Dtrace toolkit...please the last 20 lines or so (top kernel hits):

http://www.brendangregg.com/dtrace.html#DTraceToolkit

Given box is running 40% kernel time, seems that something major it going on at the kernel level.

 

[noxxsan]

hotkernel output

CPU load

 

[MarkL]

Please paste the output of iostat -exn

(btw, your cache drive is showing 120gb, not 64gb - do you have the right device in there?)

 

[packetboy]

hotkernel seems to indicate there is a fair amount of ZFS IO going on (e.g. thus indirectly resulting in lots to being put into the L2ARC and ejected as needed).

What level of IO is actually going on ?

# zpool iostat 10 10 ?

Personally, I'd benchmark the pool locally first using 'filebench' ... as obviously if that's not working well, then doing iSCISI over GigE is only going to get more complicated.

One of the most important things you need to do with iSCSI booted VMs is to ensure that you are byte-aligning the file system block size of the VM (e.g. NTFS, 4K?) with the ZFS recordsize (default 128K). Otherwise you are going to suffer IO 'explosion?'..e.g. for every 4K random IO is going to result in 128K on IO to ZFS.

Also what apps are you running on these VMs?

What's your LSI FW version?

Also, which specific SSD models are you using?

Also with regard to the power mgmt. issues suggested by others..I've had massive problems with that as well...post output of:

# powertop

 

[noxxsan]

Solved!

the issue was caused by block size, ZFS volume(512bytes) and NTFS formatting 4k block size, all ok now.

i going to use this storage mainly for Microsoft/Linux VMs, what is the recommended block size for performance?

thank you all

 

[packetboy]

How did you end up with a 512 byte recordsize initially?

Also, suggest you verify the block size requests being made by your Windows server using tcpdump and/or snoop on your iScisi traffic.

 

[noxxsan]

How did you end up with a 512 byte recordsize initially?

Also, suggest you verify the block size requests being made by your Windows server using tcpdump and/or snoop on your iScisi traffic.

napp it default block size

i had no idea that wrong block size could cause so much trouble

well, i googled a few technet articles and everyone recommend 64k block size ntfs/zfs too.