ZFS AES without AES-NI hardware support

MaDSpartus

Limp Gawd
Joined
Jun 24, 2004
Messages
158
I'm looking at either a 4 drive raidz or 6 drive raidz2 setup. I plan to run solaris natively on something basic, preferably a pentium g620, which has no AES-NI support. Can this system do software AES fast enough to saturate a 1 GB lan link? If it drops the file system performance from 300MB/s to 150, then that really wont have any practical effect over lan, and I can live with it.

Can anyone with a non AES-NI dual core shed some light? Doesn't need to be a g620, I'll find a way to correlate the results.
 
I ran some quick dd tests to an encrypted ZFS folder (AES-192) on my AMD Athlon II X2 250 (not a very powerful processor) running Solaris 11.

dd if=/dev/zero of=test bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.9705 s, 180 MB/s

dd if=test of=/dev/null bs=1M
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 34.0266 s, 31.6 MB/s

writing seemed quick but reading was quite slow...

not sure if this is helpful.

I have 16G of ram and a 60 gig ssd as a read cache, the pool where I did the test was a raid-z of 4x 2TB Hitachi 5k3000 HDD. I guess a ramdrive would have been better but I don't know how to set that up.
 
I've done a bunch of testing on this, and it's not quite what I expected.

Right now I'm running a quad core Xeon 1230, previously I had a quad core Athlon X4. Anyway, a few things to note:

- Xeon AES-NI on the E-series Xeons doesn't work for ZFS crypto on Solaris. Yes, it's supposed to, but it doesn't. There's a few other threads on here that agree if you search.

So having said that, performance on the Athlon X4 and Xeon have given me about the same performance. When using mkfile locally on the machine, I get about 140MB/sec with encryption on, and well over 500MB/sec with it off (4x five disk vdevs). This is with four threads running, with a total load on all threads at about 85%. However.... when copying a file over the network, for some reason, I don't get that same performance. I get about 70MB/sec, and it fluctuates - 120MB/sec for an instant, down to 30 or so, back up.... net is 70MB/sec. And it sits at exactly 50% load on all threads as well. There's something in the Solaris code that is causing the network and the encryption portions not to play nicely with each other.

Unencrypted doesn't show that same pattern - it sits there at 120MB/sec on a gbit link and just pegs the meter, plus or minus about 500K/sec, and only uses about 10% CPU. This is with intel cards on both ends, so YMMV with realtek or other less nice cards.

Sooo... the bottom line is if you think 70MB/sec is acceptable - for me it is. All depends on your use!


EDIT: I was talking write performance above.... reads over the network I get 60-70MB read encrypted as well, 120MB/sec solid on the unencrypted. Local reads are again much faster, in the 140-160MB/sec read on the encrypted side.
 
Last edited:
thank you very much both of you. The first post confirmed I should spring for a Xeon E-1220 due to not being able to really run encryption on a pentium. The second informed me that AES-NI doesnt even work yet on solaris. So I doubt I'll buy AES capability for the future, especially if I can dive in with a pentium for so cheap. I guess I wont be using AES for my data except except maybe some limited stuff.


thank you both
 
After thinking about this more last night, I kept digging on Oracle's web pages, and I found this:

In fact Solaris 11 isn't 'just' OpenSSL 1.0.0 but we have added our SPARC T4 engine and the AES-NI engine to support the on chip crypto acceleration. This gives us 4.3x better AES performance than OpenSSL 0.9.8 running on AIX on an IBM POWER7. We are now working with the OpenSSL community to determine how best to integrate the SPARC T4 changes into the mainline OpenSSL. The OpenSSL 'pkcs11' engine we delivered in Solaris 10 to support the CA-6000 card and the SPARC T1/T2/T3 hardware is still included in Solaris 11.

As it turns out, maybe Solaris *does* now support AES-NI, with their new Solaris 11 vice the Solaris Express 11 that I'm running - I'm going to install it over the weekend and find out. I confirmed the version of OpenSSL on Solaris Express 11 is 0.9.8 (01 June 2010), so that was prior to hardware AES support even existing, so no wonder it doesn't work. Will let everyone know how it runs on Solaris 11...
 
It "supported" AES-NI for ZFS encryption in S11 Express already, but the support in OpenSSL is a little newer.

I've tried both S11 and S11 Express on my Xeon E3 and while it seems AES-NI does in fact work with OpenSSL in S11, that is unfortunately not the case with ZFS encryption.
 
damn it, I use ZFS-encryption and AES-NI was *the* killer feature urging me to buy a nice new setup...
 
damn it, I use ZFS-encryption and AES-NI was *the* killer feature urging me to buy a nice new setup...

wallet saved i guess, simlar to me.

seems like its a desired but bugged feaure though, probably be fixed/implemented soon
 
Hmm, Oracle is showing benchmarks for ZFS AES-NI acceleration. Maybe it was finalized for Solaris 11 (Solaris 11 Express was used for one Sparc system).

http://blogs.oracle.com/BestPerf/entry/20110930_t4_zfs_encryption Has Sparc vs Xeon for ZFS encryption, although the Xeon has a huge drop in performance (3200MB/s down to 750MB/s), so it doesn't sound very optimized for Intel hardware.
 
a 2 socket xeon system is in the 700 MB/s range, bleh

I mean that is more than enough for a single socket to saturate a GB lan, but yeah not very optimized.

Well, I open it up if anyone can verify independently with a single Xeon E3/ i5-2xxx or something.
 
Darren (I think he wrote much of the zfs-crypto code) has an interesting post on his blog a couple weeks ago about how to disable AES-NI acceleration:

http://blogs.oracle.com/darren/entry/howto_turn_off_sparc_t4

For those who are benchmarking encryption you could try it both ways to see if it's actually using AES-NI and still sucks or if it's not using it at all. One thing to keep in mind that was alluded to above is ZFS encryption doesn't use OpenSSL it uses the Solaris encryption libraries.
 
Agreed. Summary I believe is:

- No openSSL support for AES-NI w/Solaris Express 11
- OpenSSL support works w/AES-NI w/Solaris 11.11.11
- AES-NI for ZFS Encryption w/Xeon E3 doesn't work on either Express or 11.11.11.

Those benchmarks were done with a "Intel Xeon Processor 5600 Sequence chips" - I think that this problem is isolated to the E3, but we need someone with one of the other Xeon series chips to test. Actually.... I could boot my Solaris image up on my Core I5 and see if it's accelerated there, as it has AES-NI as well. I might try it this weekend if I get time.
 
I wrote Darren and he instantly replied:

ZFS certainly does use Intel AES-NI because ZFS uses the kernel cryptographic framework which automatically uses AES-NI if the machine provides it. It certainly isn't broken.

Note that you can *not* turn off the kernel use of AES-NI using the methods I described so there is no (easy) way for you to disable AES-NI to show the peformance benefit of it.

What evidence (eg DTrace output) do you have to show that ZFS isn't using AES-NI ?

> Tanks for your reply. All benchmarks show there is zero difference to
> non AES NI capable processors. That's why it seems broken or better, not
> being utilized. That said the discussion is specifically about the xeon
> 1220 and 1230... I will send you some forum discussion from
> hardforum.com if you want and when I arrive at home.
> Thanks a lot for replying

Please do send me the details of how this was tested. It may just be that your tests aren't CPU bound but are IO bound which is why you don't see any difference in performance.

Can you include the output of 'isainfo -v' so that we can be sure that the Solaris kernel has actually recognised that the system has Intel AES-NI.
 
Interesting. If anyone is doing benchmarks of this, I would love if they could throw in a Windows Truecrypt benchmark as well to get some apples-apples comparison.

Anandtech has benched 2x Xeon 5670 pulling 7.5GB/s with AES-NI and Truecrypt, however this is a fully synthetic, no I/O benchmark. The fact that Solaris performance drops from 3200MB/s to 750MB/s even though the processor in theory has the capability of 7.5GB/s of encryption performance is where I got my "not optimized, kinda broken" idea.
 
I wrote Darren and he instantly replied:

Details: http://hardforum.com/showthread.php?t=1659326

my isainfo:
Code:
root@odin:/mypool/storage# isainfo -v
64-bit amd64 applications
        avx xsave pclmulqdq aes sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 sse3
        sse2 sse fxsr mmx cmov amd_sysc cx8 tsc fpu
32-bit i386 applications
        avx xsave pclmulqdq aes sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 sse3
        sse2 sse fxsr mmx cmov sep cx8 tsc fpu

EDIT: I'm using aes-256-gcm and doing encryption on folders - dunno if that matters
 
Last edited:
Here's the best evidence I can give... if Darren sends some instructions for verifying using Dtrace I can do it, but:

Using "mkfile 50g 50gfile" on an unencrypted ZFS Folder, I get this output from zpool iostat 5 (it's the same pretty much every few seconds, so I snipped it off). Basically, a little over a gigabyte per second written to the pool. CPU usage is about 10% on the Xeon.

---------- ----- ----- ----- ----- ----- -----
dozer 15.6T 20.6T 0 8.78K 51.2K 1.08G
rpool 7.19G 8.68G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
dozer 15.6T 20.6T 0 9.00K 25.9K 1.11G
rpool 7.19G 8.68G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
dozer 15.6T 20.6T 0 9.17K 0 1.13G
rpool 7.19G 8.68G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
dozer 15.6T 20.6T 0 9.08K 25.9K 1.12G
rpool 7.19G 8.68G 2 0 7.50K 0
---------- ----- ----- ----- ----- ----- -----

When I run mkfile 50g 50gfile on the exact same pool, but this time an encrypted folder, I get this:

---------- ----- ----- ----- ----- ----- -----
dozer 15.7T 20.5T 0 1.15K 409 138M
rpool 7.19G 8.68G 0 13 307 41.8K
---------- ----- ----- ----- ----- ----- -----
dozer 15.7T 20.5T 0 1.19K 0 138M
rpool 7.19G 8.68G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
dozer 15.7T 20.5T 0 1.16K 0 137M
rpool 7.19G 8.68G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----

Eight times slower. Definitely not I/O bound, either.

Output from top while doing encryption:

CPU states: 18.1% idle, 0.3% user, 81.6% kernel, 0.0% iowait, 0.0% swap
Kernel: 2178 ctxsw, 556 trap, 4345 intr, 2468 syscall, 457 flt

It's using 80% cpu just to do 130MB/sec... that's definitely NOT what is expected - I got just as good as that on the Athlon X4. I can run some numbers with truecrypt, but I guarantee it's way better than 130MB/sec...
 
Darren replied:
Try comparing between encryption=on and encryption=off but with checksum=sha256.

> is it even possible to choose anything different from SHA256 when using
> ZFS encrypted filesystems?
Not at this time it isn't. Which is why a fairer comparison is
encryption=off,checksum=sha256 vs encryption=on,checksum=sha256-mac

Anyone feeling like benchmarking?
Cheers
 
Last edited:
Hello, Darren added the following:
The following bit of DTrace should tell you if Intel AES-NI is being used or not on a given system for ZFS encryption:

It is normal to see multiple calls to the intel_aes_instructions_present function for a single call to zio_encrypt_data.


#!/usr/sbin/dtrace -s


fbt:zfs:zio_encrypt_data:entry,
fbt:zfs:zio_decrypt_data:entry
{
self->trace = 1;
printf("Start New ZFS Block\n");
}

fbt:zfs:zio_encrypt_data:return,
fbt:zfs:zio_decrypt_data:return
/self->trace == 1/
{
self->trace = 0;
printf("End ZFS Block\n");
}


fbt:aes:intel_aes_instructions_present:return
/self->trace == 1 && arg1 == 1/
{
printf("Using Intel AES-NI\n");
}

fbt:aes:intel_aes_instructions_present:return
/self->trace == 1 && arg1 == 0/
{
printf("NOT Using Intel AES-NI\n");
}
Code:
put above dtrace-code into aes-ni.dtrace
chmod +x aes-ni.dtrace
sudo ./aes-ni.dtrace
touch /tank1/crypt/foo (in another shell)

Using my old C2D 6600 DTrace traced the begin and end of ZFS blocks but did not issue any intel_aes_instructions_present, I expected it to issue "NOT Using Intel AES-NI"...
But perhaps it is because my cpu does not provide the AES-NI feature-bit, on the other hand it seems ZFS is not even querying whether I provide AES-NI :(

Cheers
 
I tried running the dtrace and it does appear that AES-NI is being used.

The output is something like:
Code:
CPU     ID                    FUNCTION:NAME
  3  37456           zio_decrypt_data:entry Start New ZFS Block

  3  73468 intel_aes_instructions_present:return Using Intel AES-NI

   ...

  3  37457          zio_decrypt_data:return End ZFS Block
The question of why encryption is so slow with Xeon E3s remains though. It is very clear that performance is CPU bound when encryption is turned on, so I would expect AES-NI to help a lot. Strangely it doesn't seem to have much effect at all.
 
Still not back from vacation to run the benchmarks until tomorrow night, but another interesting thing I see is that with four threads available (2 cores), I always use exactly 50% cpu when doing encrypted operations. So it could go twice as fast (theoretically), but CPU usage is always at 50% (viewed using top). Anyone else see the same thing? Note this is on ESXI 5 (vmware), not on the bare metal.
 
Darren replied:




Anyone feeling like benchmarking?
Cheers

I did some tests with to see the effect of sha256 checksumming and it does make a small difference, but it does not appear to be the culprit here.

This time I used a simpler pool, consisting of two Samsung 1TB HD103UJ, each as its own vdev. The system is as before - Solaris 11 on ESXi 5 with 4 vCPU. The CPU is a Xeon E3-1230 with AES-NI enabled.

Write speeds (CPU usage):
checksum=on: 192 MB/s (4-6% CPU)
checksum=sha256: 179 MB/s (~30% CPU)
encryption=on: 167 MB/s (~100% CPU)

Read speeds (CPU usage):
checksum=on: 130 MB/s (4-6% CPU)
checksum=sha256: 102 MB/s (~40% CPU)
encryption=on: 102 MB/s (~80% CPU)

According to DTrace AES-NI is being used, so it seems odd that the CPU usage should double or triple compared to checksum=sha256. These are very low speeds, considering that the CPU should be capable of >3 GB/s.
 
Any news about which CPUs support zfs AES-NI? Anyone with the new I7s like 3820 or 3930K? Want to build a new system and dont know what CPU to use..... zfs encryption is mandatory...
 
Hi,

I'm using an E5-2609 CPU on an X9DR3-F and a test install of Solaris 11 and created an encrypted ZFS on a single hard disk.

The above DTrace script says that it's using AES-NI.

Code:
# time dd if=/dev/zero of=dummy bs=32768 count=2048000
gives 589s real time and 53s sys time. ~64G file because of 32G RAM. However, top shows ~90% kernel load and ~10% idle respectively and the box draws power as if it's really working hard.

So what gives? Is it using AES-NI or not?
 
Resurrecting this thread, I have a E3-1240v2 running Solaris 11.1 and a pool with encryption set to on, and according to the dtrace listed above it is not even querying the function to test if the Intel AES instructions are present, and write performance is abysmal (~177MB/s). This is running on top of ESXi 6.
 
Back
Top