• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Seeking advice for better storage performance (Linux)

M

MountainX

Limp Gawd
Joined
Mar 2, 2008
Messages
254
Here are some examples of hdparm results on my Ubuntu Hardy server. These are from a RAID-5 array on an LSI MegaRAID 320-2E SCSI controller with 4x Seagate 15k rpm drives.

Code: 
/dev/mapper/sdX_crypt:
 Timing cached reads:   1378 MB in  2.00 seconds = 688.89 MB/sec
 Timing buffered disk reads:  166 MB in  3.01 seconds =  55.12 MB/sec

/dev/mapper/sdX_crypt:
 Timing cached reads:   1368 MB in  2.00 seconds = 683.69 MB/sec
 Timing buffered disk reads:  170 MB in  3.02 seconds =  56.37 MB/sec

This is very disappointing performance, even considering that dm-crypt+LUKS is used. I'm not sure what to try next.

My complete hardware specs:

Intel SE7520AF2 motherboard with PCI-express 8x
and 4x
ChipSet: Intel E7520 (Lindenhurst)

Two Intel® Xeon 2.8 GHz processors with an 800 MHz front side bus and 1024 KB cache.

6 GB of memory DDR2-400 MHz, ECC registered, 240-pin DIMM memory 1 GB sticks.

LSI MegaRAID 320-2E SCSI controller with:

4x Seagate ST373454LC 15k rpm drives.

Seagate Cheetah 15K.4 73GB 15K U320 80pin SCA-2 SCSI Hard Drive ST373454LC

Areca ARC-1220 SATA PCI-express 8x controller with:

2x Samsung HD103UJ (RAID-0)
2x Samsung HD103UJ (passthrough)

Motherboard SATA:
2x WD740GD Raptors
 
do you have data on that you need to save? If not try changing it to a RAID 10 array just to see what you get out of it in that configuration.
 
Lazn_Work said:
do you have data on that you need to save? If not try changing it to a RAID 10 array just to see what you get out of it in that configuration.
Click to expand...

I have a RAID-0 array on the same server and it also has performance problems. There is something else going on... I'm just not sure what it is.
 
try pulling extraneous hardware to see if that makes a difference. Perhaps there is some weird conflict in drivers between the Areca and the MegaRaid? Kinda grasping at straws though. If it were just the SCSI I'd say look into termination issues etc.. but as it is.
 
Are all your tests using hdparm? Try bonnie++ or something else. hdparm is my least favorite disk benchmark for Linux.
 
unhappy_mage said:
Are all your tests using hdparm? Try bonnie++ or something else. hdparm is my least favorite disk benchmark for Linux.
Click to expand...

I have tried bonnie++ and hdparm.

Google showed me this:
http://unthought.net/Software-RAID.H...D.HOWTO-9.html

"Bonnie++ seems to be more targeted at benchmarking single drives that at RAID"

"Few real-world programs do what bonnie does, and although these I/O numbers are nice to look at, they are not ultimate real-world-appliance performance indicators. Not even close."
 
MountainX said:
Google showed me this:
http://unthought.net/Software-RAID.HOWTO/Software-RAID.HOWTO-9.html

"Bonnie++ seems to be more targeted at benchmarking single drives that at RAID"

"Few real-world programs do what bonnie does, and although these I/O numbers are nice to look at, they are not ultimate real-world-appliance performance indicators. Not even close."
Click to expand...

(link fixed above)
That's true, but you haven't mentioned anything about real-world workloads, just synthetic benchmarks.

Are you using software or hardware raid on this box? What does the unencrypted block device perform like? How much CPU is used when testing the unencrypted device?
 
unhappy_mage said:
but you haven't mentioned anything about real-world workloads
Click to expand...
Thanks for your interest. I built this server recently. It has the fastest drives, best RAID controllers and most memory of any of my computers. (The specs are listed in my first post.) However, once I started using it, I was immediately shocked at how slow it was. Only after noticing the very slow real world file server performance did I start doing the benchmarks. They seem to confirm what I'm seeing in real-world workloads, which is performance that is significantly slower than my other computers (the ones with slower drives, less RAM, older motherboards, slower CPUs).

unhappy_mage said:
Are you using software or hardware raid on this box? What does the unencrypted block device perform like? How much CPU is used when testing the unencrypted device?
Click to expand...

1. Hardware RAID - see hardware info in my first post.
2. The unencrypted drive performs better. Cached reads are about the same, but buffered disk reads are 90 MB/s.
3. And the CPU load when testing the unencrypted device is much lower - only 1 core shows significant load.

I think part of my problem (probably not all of it) is from errors. I'm seeing two errors that look related:

1. a bunch of these are in the logs:
[104070.269918] EXT3-fs error (device dm-6): ext3_new_block: Allocating block in system zone - blocks from 78249984, length 1

device dm-6 is the RAID-0 array of 2x Samsung F1's on the Areca.

2. one of these errors is in the logs:
[ 3642.345984] gvfsd-smb[8256] general protection rip:7ff3f099e2fc rsp:423ef5b0 error:0
 
Robstar said:
How are your performance numbers w/o dm-crypt?
Click to expand...

As I was saying above, about 90 MB/s with hdparm. For example:
/dev/sdc1
Timing cached reads: 1460 MB in 2.00 seconds = 729.62 MB/sec
Timing buffered disk reads: 290 MB in 3.02 seconds = 95.89 MB/sec

Performance drops from about 95 MB/s on the drive without encryption to about 55 MB/s on the drive with encryption (same model drive, same controller, no RAID on either drive in this case -- the RAID drives are others.)

If I understand correctly, 95 MB/s from a single 7k RPM drive is pretty darn good, right?

The benchmarks I've seen indicate that performance with encryption should be within a few percent (maybe 7%) of unencrypted performance. I'm seeing more than 40% drop.
 
I know this shouldn't really be an issue for your setup, but what is your cpu usage during the bench? or just regular usage?
 
aligoodn said:
I know this shouldn't really be an issue for your setup, but what is your cpu usage during the bench? or just regular usage?
Click to expand...

Unless I'm doing heavy data transfer the CPU usage is very low. There is no process hogging the CPU. When I copy from 1 encrypted partition to another encrypted partition, it maxes out all 4 cores. When I copy from one encrypted partition to a non-encrypted partition, only 1 core shows significant load.

Thanks.
 
I think you should really investigate your network as a cause for this. Fifty megabytes per second to an encrypted device is not too shabby, but latency from your network controller, switch, or filesystem will quickly degrade your user experience. Try switching from Samba to NFS or vice versa; there's a free NFS client for Windows in SFU; it's a pain to set up but in my experience it's faster than Windows' SMB.

Also, 90 MB/s from that controller is pretty dismal. I would expect more. Perhaps try raid 10 if you don't mind reformatting.
 
You must log in or register to reply here.
Back
Top