LSR5 benchmarks

U

unhappy_mage

[H]ard|DCer of the Month - October 2005
Joined
Jun 29, 2004
Messages
11,455
A while ago there was quite a discussion going on about linux software raid 5. All I had at the time was a Highpoint 1540, which is bridged, 32/33, and generally yucky but works. Well, I finally got my Supermicro AOC-SAT2-MV8 8-port Marvell sata2 64/66 adapter going. So, it's time for benchmarks! Here are before (first line) and after (second line) and Highpoint (third line) results with bonnie++ 1.03:
Code: 
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
fs+highpoint  2000M 13138  93 35105  43 15347  25 13644  93 38291  33 190.6   3
fs+supermicro 2000M  2366  20  2707   2  1106   1  2710  19  2889   1 118.1   1
fs+marvell-bin2000M 11929  92 57600  63 24382  27 12632  91 65172  50  1985   9
longblock454  2000M  9955  77 25740  41 16422  33 15769  96 47045  45 318.2   3
lb454-faster  2000M 19581  95 37302  39 16980  13 24052  82 59534  25 275.7   1
fs-2.6.16.20  2000M 11249  88 62462  62 15159  17 13343  93 97608  60 900.6   3
lb454-sep-06  2016M 40989  93 84773  23 36543   7 37511  59 140999 12 243.8   0
The memory on this machine has also been upgraded; 256mb to 2gb :D but I disabled half of it for this test so I'd be able to run with only 2000mb size.

I noticed that it was taking a long time to run this test (like really long (as in two hours now) (final count 2:35)), so I started 'top' in another window. I found that two processes, called "ata/0" and "ata/1" were intermittently taking all the cpu time. This is with kernel 2.6.14. I believe the cause of this is new/experimental drivers in this kernel. So, I'm going to grab the latest release (2.6.14.5) and try those and see if it's any better there. Hopefully I haven't just condemned myself to sub-1MB/s writes :(

 
unhappy, please add my revised results in, I believe they are more correct for the highpoint card. Thanks.

Code: 
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
windozesucks  2000M  9955  77 25740  41 16422  33 15769  96 47045  45 318.2   3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   471  98 +++++ +++ 23255  99   463  98 +++++ +++  1832  98

Those were taken with kernel 2.6.12, many of the changes in the resent kernels should help some.
 
Changes made. I'm just looking at the STR bits for now, since that's what I felt was weaker in my setup, and more important to me since I'm storing mostly large media files on this drive.

Kernel's compiling. It'll be a while.

 
Well, the new drivers still don't support DMA on the drives :( So I'm stuck waiting for new ones.

Once DMA is out, I will strongly recommend this combination to anyone - emvs has proven reliable and flexible, and the card is only $110 for 8 sata2 ports. Can't beat that IMO. I plan to buy a port multiplier eventually, just to play with ;) My goal is 20 drives in 4 5-in-3 cages, but that's a ways off.

Holy crap, they just released 2.6.15. I may be posting again here soon.

 
Debian 3.1 stable, with my own kernels.

2.6.15 appears to have dma (woot!) but it's marked "highly experimental", so I think I'll stick with 2.6.14 for the time being, and come back in a while and see if it's not experimental anymore. Aah, who am I kidding. I'll be checking the kernel.org rss feed every 30 seconds until they release a new one. :p

Edit: feeling adventurous, I did this:
date; dd if=/dev/zero of=test.file bs=1k count=1M; sync; date
Tue Jan 3 15:22:23 EST 2006
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 20.759822 seconds (51722111 bytes/sec)
Tue Jan 3 15:22:48 EST 2006
So in 25 seconds it transferred 1GB to disk in 1k chunks. That's 40 MB/s overall. Can't complain. With 1MB chunks (1k of them) it took 19 seconds for 52 MB/s.

 
unhappy,

Using Kernel 2.6.14-1.1808 of Fedora Core 5, I have DMA turned on by default.

Code: 
[root@Garage ~]# hdparm -i /dev/hda

/dev/hda:

 Model=MAXTOR 4K060H3, FwRev=A08.1500, SerialNo=373117325445
 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
 RawCHS=16383/16/63, TrkSize=32256, SectSize=21298, ECCbytes=4
 BuffType=DualPortCache, BuffSize=2000kB, MaxMultSect=16, MultSect=16
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=117266688
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: ATA/ATAPI-5 T13 1321D revision 1:  ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5

 * signifies the current active mode

[root@Garage ~]#
 
The drivers for this card are new. It's based on the new Marvell chipset 88SX6081 (I think, there are several variations on this chipset). So the drivers are just getting to supporting DMA for that chipset now. I'm git'ing the latest sata development kernel:
Code: 
cg-clone [url]http://www.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev.git/[/url] .
and then I'll see if there are any changes. I sure hope so ;)

 
I follow ya now.

Now you have me wondering how the Highpoint compiled drivers handle the individual drives in my case. I would guess they have some type of hdparm available for tweaking.
 
I've since tried the 2.6.15 series driver. It caused some fairly bad data corruption on a small subset of my files. I recovered, and installed the Marvell binary driver. That's been working fine since. I just ran bonnie++ again, and even though the filesystem is very full now it still got better results than ever. First post updated to reflect this.

 
I've discovered some issues with my setup. Yesterday my brother got a WD3200KS and I eventually persuaded him to let me plug it into my machine for testing purposes. When I plugged it in and ran Simple Test Numero Uno - dd if=/dev/zero of=/dev/sde bs=1M count=1k. This reported only 25 MB/s - half what the 3-disk raid 5 array gets. This disk should deliver more like 60 MB/s, and in tests on other machines, it does. In other words, the card is severely limiting throughput per channel. I'm going to look into newer drivers from Marvell, as I'm using 3.41 and know there are newer ones out. Also, with these the kernel log fills up with error messages :eek:

Results will be updated in a minute.

 
Nice to see some companies actually improving their drivers. Highpoint is quick to release stable and functional drives but they don't seem to improve much beyond that! The entire time the 2220 card has been out then have only gotten to driver version 1.03!

And their open source drivers are not documented well enough for end user tweaking!

Oh well, at least they are stable, and they keep up with new Linux Kernels.
 
Update: The Linux 2.6.16.20 kernel (yeah, that's the absolute latest kernel :D) contains new code for the Marvell cards. I've been keeping track of the libata mailing list, and the driver seems to have finally stabilized. I built it, and have done a very small benchmark - dd a few gigs from /dev/zero to a file on the array, then dd the file to /dev/null. Single disks get 55 MB/s (about what I expected), and the array?
Code: 
fs            2000M 11249  88 62462  62 15159  17 13343  93 97608  60 900.6   3
See above for chart headings and what not. In summary, a lot faster reads, and a little faster writes. No complaints :D

 
Supermicro AOC-SAT2-MV8 on a Supermicro P370-DER+, with 3 300gb Maxline IIIs. 2 gigs of memory, seperate 9gb scsi boot disk. I'm finally satisfied with those read speeds; if I can get the write speeds up there I can finally sleep soundly at night :p

 
u_m, got some more! Faster CPU(3.0Ghz P4) + PCI-X (66Mhz).... Please update! I'll post the 6 disk results once I get them added in.

Code: 
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
localhost.loc 2016M 40989  93 84773  23 36543   7 37511  59 140999  12 243.8   0

Almost forgot:

Code: 
[root@localhost main]# date; dd if=/dev/zero of=test.file bs=1k count=1M; sync; date
Thu Sep 21 08:31:43 EDT 2006
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 9.76367 seconds, 110 MB/s
Thu Sep 21 08:31:59 EDT 2006
 
You must log in or register to reply here.
Back
Top