EVMS: Sadness

[unhappy_mage]

Well, I've been promoting EVMS for a while now. It's got everything in one place, and that kind of integration is Really Nice (tm) for managing disks. I've had speed problems with it, sure, but I'm using an adapter that's ill-supported by Linux. Until now.

A few days ago I downloaded the 2.6.19.1 kernel with the intention of seeing if jgarzik (or whoever else is working on the sata driver for Marvell chipsets) had added support for DMA yet. So I unplugged the data disks (all 11 of them ) and went about testing reliability. For a solid day or so I wrote a predictable pattern to a large file, and md5summed it continuously. The md5 stayed constant the entire time, so I was hopeful. I plugged 3 of the disks back in - a raid 5 array with not much on it, and nothing I cared for.

Read speeds practically didn't change at all. 20 MB/s before, 25 MB/s after. Somewhat discouraged, I decided to run some tests. Single-disk reads are about 65 MB/s, where I expect it for these disks. But reads from the evms-created raid 5 array were still terrible. So I took the plunge - copied the data off the 3-disk to the 8-disk array, and deleted the 3-disk array from evms, and deactivated the disks from evms. Then I ran "mdadm --create --chunk=32 --level=5 --raid-devices=3 /dev/md2 /dev/sdb1 /dev/sdc1 /dev/sdd1". /proc/mdstat showed it building at 65 MB/s - single-disk speeds. I'm not sure if this is a coincidence or to be expected.

An hour later (after building was finished), I ran some simple tests. "dd if=/dev/md2 of=/dev/null bs=32k count=128k" completed in about 52.4 seconds, for a transfer rate of 81.9 MB/s. For a 3-disk raid 5, that's not too shabby.

You do the math. 25 MB/s with EVMS, 80 without. My next array won't be using EVMS.

 

[drizzt81]

The linux noob wonders: what do you recommend instead? Aka: is mdadm an alternative to EVMS?

 

[unhappy_mage]

Well, it depends on your setup. mdadm does a fine job of creating raid arrays, and if you want to run your block device directly on an array that's a simple way to go. If you want a little more flexibility in your storage, you'd need to run LVM2 over top of the mdadm-created array to allow for breaking the array into chunks. LVM is a pain in the butt to set up last I checked, but it'll get the job done.

Other than using LVM+mdadm, I don't really know what there is to do. I think it's time for me to subscribe to the EVMS mailing list and fire off a message with benchmarks, and see if I'm doing something dumb. I'm using the defaults, though, and if they're not good defaults other ones should be set. I'm not using a filesystem or a volume manager, just the raw block device, so there shouldn't be anything running interference.

 

[Farva]

I can't wait to hear your results u_m.

BTW, you use Debian correct? Just a curious thought as I continue to plan my server (Most likely FreeBSD, but who knows just yet).

 

[Bones]

I never understood the reasons for using EVMS over the LVM and MD drivers. I've always used LVM on top of MD and have never had any issues. What's so good about EVMS? And don't give me the old saw about the LVM tools sucking compared to EVMS.

 

[unhappy_mage]

Yep. And that part I'm not changing any time soon - Debian seems to do things the way I like them, and I haven't had any reason to move anywhere else. apt is the nicest package manager I've ever encountered, and my experiences with yum and yast don't make me want to switch any time soon. portage is nice and all, but I've never had much luck keeping a Gentoo system in working order - they all seem to degenerate into a broken machine sooner or later.

As for testing LVM, here's what I did:

pvcreate /dev/md2
vgcreate vg0 /dev/md2
lvcreate -l143090 vg0

and then tested block-device read speed again. dd reports 80.8 MB/s on a 32G read in 32k chunks. That's more like it. EVMS, consider yourself defenestrated.

 

[Farva]

portage is nice and all, but I've never had much luck keeping a Gentoo system in working order - they all seem to degenerate into a broken machine sooner or later.

Don't let Doh see that

Ok, just PM it to her, lol

 

[Mike32]

I tried playing with evms a while ago but all the terminolgy the gui used was just confusing and really didn't make it easy to use which is stupid because the whole point of evms is to bring all the different tools together and make them easy to use. Anyway I am more comfortable doing my LVM + mdadm stuff at the command line.

Mike

 

[Dew]

Other than using LVM+mdadm, I don't really know what there is to do. I think it's time for me to subscribe to the EVMS mailing list and fire off a message with benchmarks, and see if I'm doing something dumb. I'm using the defaults, though, and if they're not good defaults other ones should be set. I'm not using a filesystem or a volume manager, just the raw block device, so there shouldn't be anything running interference.

rogue_jedi and I were doing some testing yesterday and we found that doing dd if=/dev/device was MUCH slower than doing dd if=/mnt/filename.extension
I get 70MB/sec doing:
dd if=/dev/sda of=/dev/null bs=64k count=64k

I get consistently 480-530MB/sec doing:
dd if=/mnt/mountpoint/any_random_4gb_file_here sda of=/dev/null bs=64k

Oh yes, blockdev --setra 2560 /dev/sda yeilds the best performance for me. For RJ on his LSR, it made no difference.
This is using XFS. Here are my formatting parameters:
mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=2k -l version=2 /dev/sda

I just did a little test to have numbers to back up my words:
http://decian.dyndns.org/TXStats.txt
130 Files:

Total
50141 MB copied, 107.121484 seconds,
Average
385.7 MB copied, 0.824011415 seconds, 471.7615385 MB/s
MAX
538 MB/s
MIN
282 MB/s
MEDIAN
486.5 MB/s