• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Zfs encrypted I7 CPU 100%

J

jsvensson

n00b
Joined
Mar 3, 2012
Messages
23
Hi guys,

Just built a zfs system with solaris 11 and 6x sata drives (mirrored) I use default encryption on zfs and when I copy files from my old san the transfer speed goes down every 8 seconds and then it comes back again... the cpu is quite low until it peaks to 100% every 8 seconds....check out the picture...

Screenshot%20%28800x455%29.jpg


Any idea why its like this and where to start looking?

Solaris 11
Asus Sabertooth x58
Intel i7 950 (no AES-NI)
18 gb ram
6x SATA 3TB
Intel Dual GB Nic
 
Isn't default compression on solaris 11, gzip?

And gzip is pretty slow, around 20-30MB/sec per core.

Opps, this was encryption not compression :)
 
Last edited:
I had a similar problem that slowed network transfers but it was the parity calculation on a parity zopool.
 
6x SATA drives mirrored? Ie 1 drive copied 5 times?
Or in mirror pairs?

What else you got turned on?
Compression?
Dedupe?

.
 
Like raid10...3 mirrored pairs. No dedup, no compression, smbshare on...thats it. Standard checksum...

Houk - what checksum did you use and how did you solved it?
 
Last edited:
I don't know if it's any help, but I'm seeing exactly the same thing.

enc_transfer.png


This is also on Solaris 11 with encryption (no dedup or compression). The CPU (Xeon E3-1230) has AES-NI though, which does not seem to help much. It might have something to do with the checksumming, because when I set checksum=sha256 (which is used on encrypted folders) on an unencrypted folder it also shows periodic slowdowns.
 
I did some research and it seems that either the encryption or/and the checksum is causing the bottleneck behaivour.

I wonder if there is any way to even out the peaks, i.e. having 50% cpu all the time in stead of 10% and 100% peaks....

Ill keep on searching...
 
Let us know if you find anything.

I think the high CPU usage is indeed a combination of the SHA-256 check-summing and AES encryption. It is a little strange that AES-NI is not able to accelerate the encryption part though. I briefly tried it on a Xeon E5606 at work and it had the same issues, so it's not just the E3s.

With Oracle in charge of Solaris now, I don't have expectations for a fix from their side. Instead, I hope that encryption will some day be available on Illumos. Or that some smart guy will figure out how to use a GPU to accelerate the check-summing (since it works so well with bitcoin mining, which is all about SHA-256).
 
What makes this interesting is that the real problem is not the cpu processing power - its the uneven bottleneck behaivour...its uneven. It looks like a cache issue....when the cache is full the cpu gets an uneven load...

So, the mission is to figure out exactly what is happening when the cpu rises and figure out why its not doing that all the time to even out the burden so it doesnt hit the wall...

Any ideas how to proceed?
 
Well ZFS used to "breath" like every 30seconds

ie
breathe in.... write stuff into memory..
breathe out... purge memory to drives..

Now I believe it's 8 or 5 seconds (I think 8 from memory)... which would align with your ups n downs

What are the drives dowing when the cpu tops out?.... going nuts, or idle?

ie is it ZFS writing out the checksummed encrypted data thats the bottleneck, or is it the sucking it into memory thats the hold up?

.
 
Not surprisingly, the high CPU usage occurs together with the disk writes. Nothing is written to the disks in between the peaks.

So, is there any way to make Solaris continuously write from the write cache, rather than wait until it's full and then write? Other than forcing synchronous writes of course, since that's really slow.
 
tocket said:
Not surprisingly, the high CPU usage occurs together with the disk writes. Nothing is written to the disks in between the peaks.

So, is there any way to make Solaris continuously write from the write cache, rather than wait until it's full and then write? Other than forcing synchronous writes of course, since that's really slow.
Click to expand...


My point exactly...multitasking :-P
 
Actually don't think you have said....

How are you writing the data from San to ZFS ? Ie via NFS FC or via iscsi/smb etc?

.
 
jsvensson said:
Is there any way to check what the system is doing - ie if the cache is growing?
Click to expand...

Search for arcstats.pl on google

Or if you have napp-it installed

Go to system, statistics, then arc and zil will give you some stats

.
 
I would suspect this is encryption, not checksumming (not even sha256). We've got customers throwing GB's upon GB's of data at the pool in each txg_commit and CPU does not spike much at all, and certainly not to 100%.. but of course, being OpenSolaris/Illumos-based, we don't have encryption. :)
 
You must log in or register to reply here.
Back
Top