Newbie looking to build a 10 TB server

S

Socrilles17

Gawd
Joined
Jan 12, 2001
Messages
857
Hi all, I've been building PCs for about 10 yrs, but have never really touched the raid side of things/SATA controller cards etc. My lab is generating ridiculous amounts of data and I want to build a data server for it, but don't really know where to start. I'm hoping to keep it down around $2.5k in cost is this feasible? (I can usually find pretty good prices on most PC parts). Just wondering if there are preferred 1 tb drives, is there an advantage to having the SATA ports on the mobo instead of on a controller card? What are the best controller cards for stability and where do people get their cases/any suggestions - I've been eyeballing the LIAN LI PC-A70B. I can handle everything except how to setup the hard drives, I think I want to do RAID 6 where I could recover if 2 HD's failed simultaneously. Thanks for any help, and feel free to just point me in the direction of a good tutorial if you know of any.
 
Do a search for posts by Ockie around here, he's got some pretty massive posts as well as some pretty massive servers he's built. Provides a lot of in-depth info, benchmarks, and generally everything you'd ever wanna know about massive storage goodies. 10TB... I think his last one or latest project is well into the 24TB range, or more...
 
You will have a difficult time keeping it around $2500 as the drives alone will cost nearly that much. If you want a good RAID controller, Areca is the way to go, especially if you want decent performance with RAID 6. As for cases, that's really a personal preference when you are only dealing with 10 drives, but I have a Lian-Li case for my server, and it is housing 21 hard drives at the moment (PC-V2100B). As for tutorials, well, there really isn't that much to setting up a RAID array, honestly...can't really think of any off the top of my head as I really didn't need one the first time I set up a RAID array (not to mention the manual, err, 150 page book that comes with the Areca cards covers pretty much everything).
 
I'm a relative newbie when it comes to storage as well but I think I can answer some of your questions:

I'm hoping to keep it down around $2.5k in cost is this feasible?

For RAID6 10TB using 1TB drives, the answer is no because that you'll need about twelve 1TB drives for RAID6. 1TB drives are around $190 these days so just 10 of those 1TB drives add up to $2280. However, for RAID6 using 750Gb drives, you'll need about fifteen 750GB drives for 10TB. You can get Samsung 750GB drives for $110 these days so 15 of those drives would be about $1650.

Then you'll have to figure out whether or not you want hardware RAID, hardware-assisted software RAID, or software RAID.

With hardware RAID, all of the parity calculations are done by a CPU on the card itself leaving your regular CPU free for other work. A hardware RAID will be your best bet if you want high performance out of your array. Meanwhile a hardware assisted software RAID (AKA fake RAID) is where the card sets up and maintains the RAID but the CPU does all of the parity calculations. As a result, you'll see lower performance than a true hardware card as well as slow down your CPU/system since the CPU is doing the calculations, leaving room for the server to do little else. With both hardware and hardware assisted software RAID setup, if your system dies, your RAID array will still be safe and you can easily move that array (using the same RAID card) to other systems with your data intact. However, if you're using the hardware assisted RAID chip on the motherboard itself, you'll need to find a motherboard with the exact same or similar RAID chip.

And finally software RAID is where you setup and maintain the RAID within the OS itself and all of the parity calculations done by the CPU. IIRC, you might see even lower performance than hardware assisted software RAID. Depending on which software RAID route you go, there is a chance that if your system dies, your entire RAID array with your data will go with it.

If you go the 15 x 750GB drive route, here's how the costs of the different RAID setups will play out:
Areca ARC-1260 PCI-E x8 16 Port SATA II Controller Card (Hardware RAID) - $790
2 x SuperMicro AOC-SAT2-MV8 64-bit PCI-X133MHz SATA Controller Card - $188 ($94 each, for software RAID)

Couldn't find a 16 port hardware assisted software RAID card that supported RAID6. Though I did find one for RAID5:
HighPoint RocketRAID 2240 PCI-X 16 Port SATA II RAID Card (Hardware assisted software RAID) - $425

Then you'll have to figure out if you want room for future expansion. And then figure out whether or not you want hot-swap capabilities or not. For 15 drives, you'll need three 5 in 3 hotswap bays. Each of those 5 in 3 hot-swap bays are around $100 so about $300 for 15 hot-swap bays.

As you can see, it may be possible with to have a RAID6 setup but you'll have to sacrifice performance, reliability, ease of setup and stability to do so.

is there an advantage to having the SATA ports on the mobo instead of on a controller card?
It does cut down on costs but it does limit the number of drives you can add and the performance you get. Most motherboards these days don't have more than 8 SATA ports, let alone the 10 to 15 SATA ports that you need for your setup. Also, you'll be limited to hardware-assisted software RAID, or software RAID.

If there's anything I got wrong or didn't go in depth about, please feel free to correct me. Still a relative newbie to all this.
 
10TB on raid 6, it would be impossible for that price range.
 
"My lab is generating ridiculous amounts of data and I want to build a data server for it"

The lab did not just suddenly generate ridiculous amounts of data. I suspect you have a reasonable amount of data on hand and are producing a steady stream of data and at some point int he future you expect to have 10TB of data stored.

The question for or against RAID is how fast do you need to access the data. 70MB/sec is about the speed that a non-RAID hard drive will transfer data. That might be good enough for you.

The cost is easy to figure out. I would disagree with the costs posted. I would look for a design that could be expanded.
 
I would hate to argue with Ockie, but its never impossible. Especially with Ebay.

Even if you go the Areca route ( which is what I would go if it was my own setup ), it could work out to:

$750 for controller
$1594 for hard drives ($99 / each for 750gb seagate sata pricewatch special @ memorylabs)
$99 for 750w PSU to power those drives

total = $2337 if you can avoid the CA tax

16 x 750gb in raid 6 would be 10.5TB .. alas, it wont be 10TB after formatting though so maybe okie is right after all =T
 
I meant to say 8 gb of total storage space (10 TB of drives). I would like to thank every1 for the helpful advice so far. As far as prices go, I can usually find some crazy deals (example I was able to get 500 gb HD's for $40 a piece last year) and there is a deal that just popped up where I might be able to get 1 TB drives at $150 a pop (complicated though). One last question where do you shop for computer cases that hold 15+ drives?

thanks again for all the help - oh and we will be doing software raid as speed is not really important.
 
Socrilles17 said:
I meant to say 8 gb of total storage space (10 TB of drives). I would like to thank every1 for the helpful advice so far. As far as prices go, I can usually find some crazy deals (example I was able to get 500 gb HD's for $40 a piece last year) and there is a deal that just popped up where I might be able to get 1 TB drives at $150 a pop (complicated though). One last question where do you shop for computer cases that hold 15+ drives?

thanks again for all the help - oh and we will be doing software raid as speed is not really important.
Click to expand...
Any case should do. Just get these: iStarUSA BPU-350SATA Aluminum Black 3 to 5 Bay HDD Hot-Swappable SATAII Enclosure - Retail - $109.99 each
 
Socrilles17 said:
One last question where do you shop for computer cases that hold 15+ drives?
Click to expand...

Any case with nine 5.25" bays can hold up to 15 drives using 5 in 3 hot-swap modules like the one AMD_RULES posted. Or if you don't care about hot-swap, you can use these dumb 5in3 drive modules:
3-to-5 Mounting bracket - $29

So if 15 drives is your max limit, then any case with nine 5.25 drives will do. If you want something with a bit more room, you can get this Stacker case:
COooler Master Stacker 810 RC-810-KKA1-GP Full Tower Case with RS-850-EMBA 850W Power Supply - $270

With that case you can hold up to 20 drives using 5in3 drive modules.
 
When people add drives to the 5.25" bays do you usually add cooling?, most the cases seem to come with cooling where the internal HD mounts are, and this would be lacking in the 5.25" I know there are mounts you can buy with cooling, but I'm just wondering if people think it necessary or not.

Thanks again
 
Socrilles17 said:
When people add drives to the 5.25" bays do you usually add cooling?, most the cases seem to come with cooling where the internal HD mounts are, and this would be lacking in the 5.25" I know there are mounts you can buy with cooling, but I'm just wondering if people think it necessary or not.

Thanks again
Click to expand...

They usually do from what I've seen so far. Usually the hot-swap bays come with their own fans so the cooling needs are taken care of there. With that many drives crammed in a such a small space, I highly recommend cooling them.

BTW, this hot-swap bay uses 90mm sized fans:
Supermicro CSE-M35T-1B Black 5 Bay Hot-Swappable SATA Backplane - $100.

90mm fans > 80mm fans.
 
The 5in3 and 4in3 have decent cooling fans. The cooler master centurion and stacker cases are good options for this type of build(pretty much top to bottom 5 1/4 bays).

Depending on the scope of the project dual PCI-E cards from either Areca or 3-Ware should handle the amount of internal connectors needed. I've had good results with moving raid arrays between 3ware raid controllers in the past, so swapping an 8 for a 12 port would be an option.
 
You do realize that when your budget, homebrew setup loses data you will be blamed and could possibly lose your job? I work in this industry and I have seen it happen several times! Management does not understand technical problems; especially technical disasters. For the sake of self-preservation alone you should go with a vendor that will provide support. The organization you are with should be willing to spend 8-12k on a decent setup, otherwise you should start looking for a new job where business risks are dealt with properly.
 
I'm in academics, and I can't stand the amount of money we pay for crap, ie Apple servers which are over priced 10X, for example, a hard drive crapped out on one of our macs and when we took it in for repair we were charged $400 for a replacement 80 gb hard drive, this was a 4 year old computer, which we could have replaced with a faster cheaper PC then the stupid replacement hard drive. But realize I am in a lab that is like 5 people a professor 3 grad students and a couple undergrads. As long as I can build something stable (RAID 6) at a reasonable price I will not be blamed if something goes wrong people understand that computers have problems. Oh and since stability is important corners will not be cut to save costs. I will only buy quality parts I just came in thinking $2000 to 2500 will be enough, if its more its more.

On a positive note I was just able to pick up e8400's at $140 a pop and will use one of them in this build, does any1 have a recommendation for a motherboard that can use this (1333 fsb) processor that could handle the two SATA controller cards?
 
Will the SATA controller cards be PCI-E, PCI-X or PCI?
 
Dont you also have to try and factor in a backup solution, which could a fair chunk also. Do not rely on RAID6 to provide you with a backup, its for uptime only :)
 
nefarious986 said:
I would hate to argue with Ockie, but its never impossible. Especially with Ebay.

Even if you go the Areca route ( which is what I would go if it was my own setup ), it could work out to:

$750 for controller
$1594 for hard drives ($99 / each for 750gb seagate sata pricewatch special @ memorylabs)
$99 for 750w PSU to power those drives

total = $2337 if you can avoid the CA tax

16 x 750gb in raid 6 would be 10.5TB .. alas, it wont be 10TB after formatting though so maybe okie is right after all =T
Click to expand...

The drives, psu, and controller are the easy and cheap ones, it's the little things that eats you alive.

You'd want a BBU, $100+, you want a case to hold those drives, you are at least looking at $150 for the case and possibly an additional $300 for the drive units to maximize your case space. Motherboard, CPU, and Memory + System drive, at least $200. Hopefully you don't need any cables as breakout cables are $30-50 each.

Now factor that with the reasoning that you used 750gb drives, which would be less than 10tb after formatting, so you'd realistically need an additional or larger controller and more drives.

Price keeps climbing. Add shipping and taxes and stupid little things that you will end up needing and it even climbs higher.

You can ebay things, this is basically what I am a pro at, but keep in mind, a lot of times you get what you pay for. :(


I would feel comfortable with a $3,500 budget for these requirements, then you can perhaps get a little better hardware out of the deal and get more piece of mind,.
 
Did you consider looking into Unraid from Lime Technology?

http://lime-technology.com/

I use it for a few different customers and have had no problems. It's easy to expand later, and very easy to replace hard drives, you don't even need the same size or type of drives.

It's worth considering if you are on a tight budget. You could setup a 5TB system now and just add hard drives as your space needs increase.

Actually, after reading the whole thread again, I think unraid will be perfect for your situation. I won't list all the advantages, I am sure you can read the site :) but if you got any questions I will be happy to answer.

Another solution is a disk storage system like this

http://www.norcotek.com/item_detail.php?categoryid=8&modelno=DS-1220N

I purchased this for one of my customers and they are currently using 8TB's of data.
 
Again thanks for all the help and suggestions every1!!!

So far I've picked up a mobo:
The intel badaxe d975bx2 (8 SATA connections so I think I will try software raid and see how it goes)

case:
Rocketfish case from bestbuy (lian li)
going to pick up a 3X5.25 to 4X3.5 converter from coolerguys

I also got some e8400's pretty cheap unfortunately they dont fit in the mobo so I'm going to try to trade one of them for e6700 or e6800

As far as OS goes I have a friend who knows a lot about linux so I'm going to let him figure out the software side of things.
 
You must log in or register to reply here.
Back
Top