RAID 5 Performance with dm_crypt

H

hastala

n00b
Joined
Aug 8, 2008
Messages
6
Hi,

i've freshly installed latest Ubuntu (Hardy Heron) to my new PC and i'd like to find out if the RAID 5 performance i get is sufficient.

My system:
AMD 4850e (2 x 2,5 Ghz), not OC'd
Asus M3A-H/HDMI

My RAID 5:
4 x Samsung HD642JJ, connected directly to the mobo (using AHCI option)
64k chunk size
NCQ enabled (for more options, see below)
encrypted with luks and dm_crypt

HDPARM on the RAID:

Code: 
/dev/md0:
Timing cached reads: 1494 MB in 2.00 seconds = 746.93 MB/sec
Timing buffered disk reads: 630 MB in 3.01 seconds = 209.49 MB/sec

bonnie++ using a 3,4GB file:
my.php


do you think these values are rather good or rather bad?
 
The hdparm run is a little on the low side, maybe, but not out of line with expectations for a non-encrypted RAID5... which is what you were testing. With encryption, you would be doing well to get half that. Since you didn't post the bonnie command used for that run, we have no idea what you're testing, so it doesn't mean a whole lot. Post some terminal input/output, not a jpeg please.
 
This is hdparm for the encrypted volume, looks like the reads are really bad now, can that be normal for the encrypted volume?

Code: 
root@godfather:/content# hdparm -tT /dev/mapper/md0_crypt

/dev/mapper/md0_crypt:
 Timing cached reads:   1544 MB in  2.00 seconds = 772.35 MB/sec
 Timing buffered disk reads:  198 MB in  3.01 seconds =  65.73 MB/sec

And here's the bonnie++ output for a 3,4GB file, forgot to mention i'm having 2GB RAM in my machine:

Code: 
root@godfather:/content# bonnie++ -u root -d /content
Using uid:0, gid:0.
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.03b       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
godfather     3520M 47094  76 63277  22 33435   9 63663  88 92929  15 402.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
 
i did one more test, so here's hdparm -tT of the relevant drives for comparison:

1) single drive of the RAID 5 array
Code: 
/dev/sdb:
 Timing cached reads:   1524 MB in  2.00 seconds = 762.25 MB/sec
 Timing buffered disk reads:  344 MB in  3.01 seconds = 114.17 MB/sec

2) unencrypted RAID 5 array
Code: 
/dev/md0:
Timing cached reads: 1494 MB in 2.00 seconds = 746.93 MB/sec
Timing buffered disk reads: 630 MB in 3.01 seconds = 209.49 MB/sec

3) encrypted RAID 5 array
Code: 
root@godfather:/content# hdparm -tT /dev/mapper/md0_crypt

/dev/mapper/md0_crypt:
 Timing cached reads:   1544 MB in  2.00 seconds = 772.35 MB/sec
 Timing buffered disk reads:  198 MB in  3.01 seconds =  65.73 MB/sec

number 2 shows what i had expected, a large performance boost from the raid array, but the performance drop on the encrypted volume seems pretty enormous?! read in some article that 5-10% decrease could be expected...

cpu usage when writing blocks was 22%, so i guess that's no bottleneck...
 
Your CPU usage is likely wrong, I'm guessing you watched top? It's not accurate, especially for short runs like hdparm. Also, the actual encryption is handled by a kernel thread, not hdparm itself, and is reported as such for CPU time.

Encryption is definitely a bottleneck, but the degree depends on the algorithms used. 128-bit aes-cbc is well-optimized and lightweight compared to something like 256-bit twofish, for example. And if you go with something other than cbc, like cmc or lrw, you can expect performance to degrade further.

I have a test setup here that hdparm reads at 103 MB/sec unencrypted, but reads at about 45 MB/sec with - this on a Q9450 at 2.66 GHz (stock speed).
 
22% cpu usage i mentioned was while bonnie++ wrote 3,4GB in blocks (see my 2nd post for the whole bonnie++ output)

i'm using 256bit aes-cbc, here's how i encrypted the array:
Code: 
cryptsetup -c aes-cbc-essiv:sha256 -y -s 256 luksFormat /dev/md0

but your test setup is not in a software raid, rite?

is there anybody who can share his software raid hdparm output (without encryption), so i can at least compare this to mine?
if somebody has also encrypted his software raid and would like to show off his values, you're very welcome, too ;)
 
Here's one for RAID5, 4 x Western Digital 640s:
Code: 
# hdparm -t /dev/md0

/dev/md0:
 Timing buffered disk reads:  738 MB in  3.01 seconds = 245.40 MB/sec
That's from the RAID machine, unencrypted of course. Unfortunately, I don't have crypto compiled into the kernel on the RAID-5 box, but I do on a different machine. So I carved a small logical volume out of the RAID, exported it via ATA over Ethernet, and set up crypto from a different box (one that has crypto in the kernel).
Code: 
# hdparm -t /dev/etherd/e2.0

/dev/etherd/e2.0:
 Timing buffered disk reads:  296 MB in  3.00 seconds =  98.51 MB/sec
# cryptsetup luksFormat -c aes-cbc-plain:sha256 -s 256 /dev/etherd/e2.0
# cryptsetup luksOpen /dev/etherd/e2.0 ctest
# hdparm -t /dev/mapper/ctest

/dev/mapper/ctest:
 Timing buffered disk reads:  238 MB in  3.01 seconds =  78.95 MB/sec
The first hdparm run is on the exported RAID volume, obviously limited by the gig Ethernet pipe between the test box and the RAID box; about 100 MB/sec is typical for AOE setups. The second run is on the encrypted volume, and is considerably slower.

My test box is a little faster than yours, but it's clear that disk speed is not the bottleneck here, CPU is.

One last benchmark, this time from the local machine, 1 x Western Digital 640 (The logical volume I used for the testing was at the start of the disk):
Code: 
# cryptsetup luksClose ctest
# cryptsetup luksFormat /dev/wd640/crypt-test -c aes-cbc-plain:sha256 -s 256
# cryptsetup luksOpen /dev/wd640/crypt-test ctest
# hdparm -t /dev/mapper/ctest

/dev/mapper/ctest:
 Timing buffered disk reads:  258 MB in  3.00 seconds =  85.97 MB/sec
Without the AOE related network overhead, performance goes up a little bit - 6 MB/sec (about 8%). Unencrypted reads are 105 MB/sec.
 
You can try stuff like a different cipher (twofish/salsa20) hash (whirlpool) and if your on a dual/quad core, put the encryption under the raid. That's what I do personally.

Also, instead of cbc try lrw.

Note that some of that is experimental in the late kernels, so you could run into trouble.

I got cryptsetup directly on the drives, and mdadm assembles /dev/mapper/encrypted-raid1-5 into /dev/md0. Allows kcryptd to be run 5 times, which increases performance very considerably.
 
@Bones: wow, thanks for the extensive testing!!!

these WD 640, are they better than Samsungs HD642JJ? I see your RAID is performing slightly better that mine... is this software raid or do you have a real HW card for that?

@NeuronN: i wouldn't like trying something that you say is experimental, the data on the RAID is quite important to me (yeah, i do have backups, but only of the most important stuff).

but i'd like to hear more about the encryption under the raid, i'm having an AMD X2 64 @ 2,5 Ghz (4850e).
what about failing disks, can mdadm re-assemble the array if the disks instead of the whole array are encrypted?
i read somewhere that having a raid encrypted (either way) is kind of critical, when having a power outage, do you use a UPS for your raid?
do you know of any downsides, compared to having the whole array encrypted?
 
You must log in or register to reply here.
Back
Top