My do-it-myself RAID5 project

I

idea

Gawd
Joined
Jan 24, 2005
Messages
615
I need to build a RAID5 machine from scratch. I've been reading about them the whole weekend but I can't decide where to begin a base for building. Whats the advantage between onboard RAID5 and a PCI card RAID5? Since its a personal project, cost IS an issue. However this is certainly not a game where we play "as low as we can go." Whatever the cost to be done right.

What I need (aside from common features): 6 to 8 SATA ports for future expandability. Over 1TB in RAID5. Decent gigabit speeds. Linux compatibility.

I've been browsing on newegg, and these are my 2 options, any way I look:

$100 Any AM2 ATX w/ onboard everything PLUS 6 port SATA RAID5
$50 AMD Sempron64 CPU
$50 512mb DDR2 800
---
$200.00
Click to expand...

$75 Any AM2 ATX w/ onboard everything (but no 6 port RAID5)
$50 AMD Sempron64 CPU
$50 512mb DDR2 800
$250 PCI 8 port SATA RAID5 (software)
---
$425.00
Click to expand...

As you can see, the convenience of having 6 ports onboard cuts your cost by over $200.00. Is it worth it? The PCI card has comforts like LED lights to show failures, and the ability to transfer to another motherboard combo, but I dunno if its worth the cost.

Then I still have to buy a case/PSU. If anyone can recommend a nice case for this cause, I'm all ears. Right now I'm looking at that Free After Rebate mid-tower and quiet PSU deal from Frys.com. It's cheap crap but it'll do.

And FYI, only one Intel board offers 6 port RAID5. Its mATX which I like, so I may go with that one. I was just comparing AM2's for reference.

The hard drives, well, thats the last step. There were a few 400GB for $100 deals this week, I think I'll buy 4 or 5 to start off with.
 
I just built two fileservers for cheap, they are supported excellent under Linux with hot swap functionality and all. Dual gigabit ports, pci-e expandability... its pretty good.

A8N SLI Premium (forcedeth+skge for nic's, sil3114+nv_sata = 8 port goodness)
A64 3200+
2x256mb DDR400
80gb IDE drive for O/S
Some generic case/power supply

Total cost was like $420 or something shipped. I did price analysis for about 2 weeks before settling on this setup, and it's pretty awsome. 4 disk raid5 was getting about 60mb/s writes and about 90mb/s reads, 4 disk raid0 was about 110mb/s writes and about 120mb/s reads.

Pictures @ http://www.hoka.org/gallery2/main.php?g2_itemId=6308 if you're interested.

3200
7200.9 80gb
A8N-SLI Premium
Generic Case
RAM
 
you may have trouble finding a matx motherboard with a good integrated raid controller.
You may want to invest more into a Highpoint raid controller.
 
idea said:
ahh, 939! For some reason I didn't think about that. Thanks!
Click to expand...

Yeah, theres a few things you should be aware of with these boards if you are going to run Linux. First off, raid5 expansion requires a newer kernel (>=2.6.17 iirc), and a newer version of mdadm (>=2.5.2). If you want hot swap functionality, the silicon images chipset supports it out of the box. The NVidia chipset requires a hard-port-reset patch which is available in the 2.6.19 tree, I had a bunch of issues with 2.6.19 as a whole and therefore it didn't really work out. Its a like 3 line patch thats very easy to apply by hand, so you can manually put it on anything really. AHCI does not work on this version, you will have to use sil/sata_nv drivers. Also, the nvidia gigabit chipset is a bit better than the marvell, the nvidia chipset uses about half as much CPU as far as I can tell, no idea why.

The blue pci-e slot (video) is video only! You can't put anything in there, you can however leave it empty, manually switch to dual video card (for the 8x8x split as opposed to 16x1x) and put a raid/nic whatever in the second slot. Theres also a pci-e 4x slot which only has 2x wiring, but I doubt you'll need that much bandwidth anyways. As a whole, support is excellent and everything basic works out of the box (at least for Gentoo). I'd also highly suggest some udev rules to associate a port with the different sd* block devices, that way you know which block devices corresponds to which sata port (and therefore which drive).

Code: 
cat /etc/udev/rules.d/10-local.rules
KERNELS=="0:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/1d%n"
KERNELS=="1:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/2d%n"
KERNELS=="2:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/3d%n"
KERNELS=="3:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/4d%n"
KERNELS=="4:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/5d%n"
KERNELS=="5:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/6d%n"
KERNELS=="6:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/7d%n"
KERNELS=="7:0:0:0", SUBSYSTEMS=="scsi", SYMLINK+="raid/8d%n"

ls /dev/raid -l
total 0
lrwxrwxrwx 1 root root 6 Dec 17  2006 1d -> ../sda
lrwxrwxrwx 1 root root 7 Dec 17  2006 1d1 -> ../sda1
lrwxrwxrwx 1 root root 6 Dec 17  2006 2d -> ../sdb
lrwxrwxrwx 1 root root 7 Dec 17  2006 2d1 -> ../sdb1
lrwxrwxrwx 1 root root 6 Dec 17  2006 3d -> ../sdc
lrwxrwxrwx 1 root root 7 Dec 17  2006 3d1 -> ../sdc1
lrwxrwxrwx 1 root root 6 Dec 17  2006 4d -> ../sdd
lrwxrwxrwx 1 root root 7 Dec 17  2006 4d1 -> ../sdd1

Good luck!
 
hokatichenci said:
I4 disk raid0 was about 110mb/s writes and about 120mb/s reads.
Click to expand...
For a four disk RAID-0 array those numbers appear awfully "slow".
 
Say I were to go with a mobo with onboard 4 port RAID5, and in the future i wanted to expand. What is the process of taking 4 existing HDDs with RAID5 and switching controllers? I would have to transfer the data on another system first? Or is the RAID5 system some kind of standard between all controllers?
 
idea said:
Say I were to go with a mobo with onboard 4 port RAID5, and in the future i wanted to expand. What is the process of taking 4 existing HDDs with RAID5 and switching controllers? I would have to transfer the data on another system first? Or is the RAID5 system some kind of standard between all controllers?
Click to expand...

There is no standard, also, you will not find a motherboard with onboard hardware sata raid. They're all BIOS (read: software) raid, and will not function that way under Linux. If you use Linux's software raid, you can switch controllers, ports, or even systems so long as the md versions are compatible. mdadm (by default it seems) uses the UUID (disk identifer) to determine arrays, so you can switch things around so long as the disk is still there according to the kernel.
 
Yes, I understand that. I like that using mdadm to manage your disks makes your RAID "universal" (since linux can be applied to almost any situation) but I feel as though RAID should be something configured through a BIOS type menu. Do you understand what I mean?

Anyway, I know that if I used softraid (linux, windows, freenas, etc etc) it would save me a bundle but I don't trust the OS to handle the task.
 
idea said:
Yes, I understand that. I like that using mdadm to manage your disks makes your RAID "universal" (since linux can be applied to almost any situation) but I feel as though RAID should be something configured through a BIOS type menu. Do you understand what I mean?

Anyway, I know that if I used softraid (linux, windows, freenas, etc etc) it would save me a bundle but I don't trust the OS to handle the task.
Click to expand...

The OS handles the task in a BIOS raid setup too, its just that with Linux they choose to ignore the BIOS completely because it's totally useless. If you want real hardware, you will have to pony up at least $250-300, the 8 port Areca card is about $500. You are going to have to put faith into whatever system you use, with a proprietary hardware raid you can't necessarily move between systems, and if your controller goes you need a backup controller or a redundant setup. With OS raid (windows dynamic disks or linux md or whatever bsd uses) you can at least move between controllers, setups, and pretty much anything. Sure, things could screw up, but that's what backups are for. If you put a battery backup on your raid server with an automatic shutdown script if power goes for more than say 10 seconds you'll be far less at risk. The important thing here is that you want to do it for cheap, to get truly high available/safe system you are going to have to pony up a ton of cash.
 
I'm doing some more reading on allowing Linux to manage the array, here's an interesting paragraph when it comes to these "cheaper" controllers

http://linuxmafia.com/faq/Hardware/sata.html
Linux often cannot read existing fakeraid volumes on such host adapters, unless you're willing to use proprietary fakeraid drivers (where available). But unless you're dual-booting MS-Windows, you shouldn't care, because Linux's software RAID (kernel "md" driver) is much faster and more reliable. You're advised to blow away fakeraid volumes, use SATA drives as straight block devices, and enable Linux software RAID instead, during Linux installation.
Click to expand...

So, if you're not gonna spend $500 on a hardware RAID card, don't spend anything. Now I'm off to find the best deal on a motherboard with 6-8 SATA ports. I guess it doesn't matter if they're all SATA I or II, I doubt the config will ever be going past 1.5GB/s anyway.

I'm pretty good in linux. I don't trust myself to manage a disk array, but I'll suppose I have to learn because its worth saving $500. God help me if something goes wrong, a disk goes bad, whatever. I know one day I'll screw it all up.
 
idea said:
So, if you're not gonna spend $500 on a hardware RAID card, don't spend anything.
Click to expand...

Pretty much, you could try stacking different SATA cards but that probably wouldn't work out too well. You will want to make sure that whatever chipsets you go with that you have solid support for them. If you go with Linux you will definitely want to read this:
http://linux-ata.org/driver-status.html

Stay away from the ATI chipsets (I know that DFI board with 8 sata ports looks nice though!) because they are very poorly supported, if at all under Linux.
 
Right, I need to find a motherboard with AHCI SATA. Thats the most Linux compatible.

Driver name: ahci

Summary: Full NCQ support, full SATA control including hotplug and PM.

Note1: AHCI specification is completely open.

Note2: ATI, Intel, JMicron, NVIDIA, SiS, ULi and VIA are currently known to have deployed AHCI in their chipsets.

Hopefully others will follow. AHCI is a nice, open design.
Click to expand...
 
Yeah, AHCI is open standard which is nice and pretty, but only really new chipsets support it. If you are looking for a cheaper solution it might not work out, do keep us updated on what hardware you decide on and how you set it up. I'm curious as to how it will compare to my systems :cool:
 
idea said:
http://linuxmafia.com/faq/Hardware/sata.html

So, if you're not gonna spend $500 on a hardware RAID card, don't spend anything. Now I'm off to find the best deal on a motherboard with 6-8 SATA ports. I guess it doesn't matter if they're all SATA I or II, I doubt the config will ever be going past 1.5GB/s anyway.
Click to expand...

A highly unbiased source, which names itself accordingly. I'd take their claims 100% at face value -- their implementation is far superior to every sub $500 RAID controller.

(And the small matter of the sub-par performance mentioned in this thread is surely the poster's fault in some way.)

/sarcasm.

While there's much good to be said of the Linux RAID implementation, I think they've gone too far if such claims are to be believed.
 
hokatichenci said:
4 disk raid5 was getting about 60mb/s writes and about 90mb/s reads, 4 disk raid0 was about 110mb/s writes and about 120mb/s reads.
Click to expand...

Ouch. That sucks.

My RR2320 gets
R0, 4 drives:
Read: 244MB/sec
Write: 160MB/sec

R5, 4 drives:
Read: 199MB/sec
Write: 78MB/sec
 
Yeah, software raid (especially r3/5/6) is by no means fast. If you need performance, you are going to need to dump money into a fast raid card. My 8 disk raid6 Areca array gets like 140-220mb/s writes. But if you want cheap - software raid is completely free so long as you've got the free ports.
 
Funny thing is that the RR is disparagingly called "fakeraid" by the Linux Mafia -- yet it seems to trounce the Linux performance here.

Don't get me wrong -- I wouldn't really take these two independent performance reports as a fair comparison. I think the only fair comparisons would be done, by several different people on several different computers, with Linux vs. others on the same boxes with their best efforts. I'm just pointing out that some of this material, taken as absolute, is incredible.
 
How many users you planning on running on this? If you are going to run less then 5 users then software raid 5 under linux is fine. If you want to run more or need the extra speed for whatever reason starting around 250 to 300 can can pick up raid cards with onboard cache that start to really show a jump in preformance in many cases. Mind you the best results will come with much more expensive controllers but I'm thinking this is going to be a personaly system.
 
Madwand said:
Funny thing is that the RR is disparagingly called "fakeraid" by the Linux Mafia -- yet it seems to trounce the Linux performance here.

Don't get me wrong -- I wouldn't really take these two independent performance reports as a fair comparison. I think the only fair comparisons would be done, by several different people on several different computers, with Linux vs. others on the same boxes with their best efforts. I'm just pointing out that some of this material, taken as absolute, is incredible.
Click to expand...

Yeah, I don't take any of those comparisons very seriously. I didn't even read the link that was supplied but software raid is by no means a replacement for hardware raid - they each have their places. Right now since I'm playing with raid across multiple systems, saving say $100 on a single system means saving $200 on 2 systems, $1000 on 10 systems. Correct raid cards for the job would just be too costly. Doubt the OP is going my direction but cheaper is definitely nicer ;)
 
idea said:
Then I still have to buy a case/PSU. If anyone can recommend a nice case for this cause, I'm all ears. Right now I'm looking at that Free After Rebate mid-tower and quiet PSU deal from Frys.com. It's cheap crap but it'll do.
Click to expand...
Please, do not go with a cheap PSU, get something of quality. You don't want to fry your mobo, drives, etc with poor power. Or even not having enough amperage to boot everything up. It sickens me that you would spend all this money on parts and yet use something that will probably kill your machine.
 
hokatichenci said:
Doubt the OP is going my direction but cheaper is definitely nicer ;)
Click to expand...

I beg to differ! I commend you for your research on hardware. I played with newegg motherboard configs for a whole day and each time the A8N-SLI Premium was the only one with greater than 4 ports and a linux compatible chipset. The aging 939 socket means I can throw in some relatively cheap hardware, then upgrade in the future.

For the record this is just a personal fileserver, no more than 5 users at any time, LAN or WAN.
 
protias said:
Please, do not go with a cheap PSU, get something of quality. You don't want to fry your mobo, drives, etc with poor power. Or even not having enough amperage to boot everything up. It sickens me that you would spend all this money on parts and yet use something that will probably kill your machine.
Click to expand...

Noted ;)
 
As promised, here's a thread update. I finally got around to purchasing all the parts needed.

Intel P4 631
Intel DG965 (onboard everything including 6 SATA on ICH8 chipset)
ALPINE 7 cooler
2x 256mb DDR2
4x WD 320gb RE2
1x WD 80GB
The case will be an Antec 2650 that I had lying around with a great 5-drive cage and 120mm fan.

Total cost of the parts WITHOUT drives is $260 :cool: Some parts were purchased from the trade forum, thats how its so cheap.
 
hokatichenci said:
The blue pci-e slot (video) is video only! You can't put anything in there, you can however leave it empty, manually switch to dual video card (for the 8x8x split as opposed to 16x1x) and put a raid/nic whatever in the second slot. Theres also a pci-e 4x slot which only has 2x wiring, but I doubt you'll need that much bandwidth anyways.
Click to expand...

So, with the A8N32 I can only plug a 4x PCIe RAID card to the 4x slot (if at all possible because of the heat pipe) or to the black "16x" slot? Could I then use a graphics card on the blue slot running at 8x and still use the black slot below with a raid card there running also at 8x?

I'm considering either getting a 8 ports Highpoint card now and add a second one in the future or if this would not be possible with my board get a 16 ports card right away.
 
smartins said:
So, with the A8N32 I can only plug a 4x PCIe RAID card to the 4x slot (if at all possible because of the heat pipe) or to the black "16x" slot? Could I then use a graphics card on the blue slot running at 8x and still use the black slot below with a raid card there running also at 8x?

I'm considering either getting a 8 ports Highpoint card now and add a second one in the future or if this would not be possible with my board get a 16 ports card right away.
Click to expand...

Yes, you can put a 4x PCI-e card into a 16x PCI-e slot. As for running multiple card, I'm not sure.
 
Im running way too much hardware for a simple fileserver, but until the gaming box gets built its also my primary gaming machine, so its not a HUGE waste at least.

Running an A8N-E for the board. While it does have onboard SATA ports and raid functionality, the only things on it are the optical and the boot raptor.

The actual raid card (software raid, actually) is the 2320. PCI-E x4 8 port R5 setup. Fast, simple. If I need to I can pull the whole setup and swap it to a new machine. You cant do that with onboard stuff.

</rambling input>
 
Why do you say you are running way too much hardware? Your description sounds fine. Did you leave something out?

Since Linux will be managing the RAID, I can certainly swap hardware. To migrate a RAID to a new machine I just need to be sure I will be using Linux on it. The onboard SATA RAID controller's BIOS will never need to be configured.
 
weee... i got it set up tonight. everything went ok except for a few things i'll iron out later...

# mdadm -D /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Mon Jan 29 08:08:22 2007
Raid Level : raid5
Array Size : 937705728 (894.27 GiB 960.21 GB)
Device Size : 312568576 (298.09 GiB 320.07 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Mon Jan 29 13:41:01 2007
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 32K

UUID : f289a0e5:511feb7b:b0489ce9:ac8c8d6c
Events : 0.30

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
Click to expand...

What kind of speeds should I be getting? How do I check it?
 
idea said:
weee... i got it set up tonight. everything went ok except for a few things i'll iron out later...



What kind of speeds should I be getting? How do I check it?
Click to expand...

Maybe hdparm -Tt /dev/mdX
or mount a filesystem on the md device and run bonnie++
 
hdparm only works on individual block devices, IIRC. You'll need to figure out bonnie++, not that that's terribly hard. Make to use about 10 times your ram as the test size to avoid the effects of memory caching.
 
Here you go

Code: 
$ bonnie++
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
santo            1G 25510  97 51847  10 20153   2 30638  79 47945   3 227.0   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  4662  99 +++++ +++ +++++ +++  3894  98 +++++ +++ 15088  99
santo,1G,25510,97,51847,10,20153,2,30638,79,47945,3,227.0,0,16,4662,99,+++++,+++,+++++,+++,3894,98,+++++,+++,15088,99
 
You must log in or register to reply here.
Back
Top