any suggestions for 48TB+ server

T

tgrimley

Limp Gawd
Joined
May 1, 2006
Messages
311
Is it possible to do this with $10k?

I need about 24TB for storage of recorded data, and about another 24TB to play with the data. I have used linux software raid for a number of years. Hardware raid sounds more attractive in this build, but I don't think it could be in budget.

Anyone have any suggested builds?

thanks in advance!
 
How about a SM SC847 36 Hotswaps - $2500
SM X8TH-6F - $600
24GB DDR3 ECC RAM - $800
24x 2TB HDDs - $4080
2x Intel E5530s - $1100
2x Intel CPU coolers - $60
Areca 1680X - $400
HP SAS Expander - $200

Total - ~$9700

You can even save money by getting not as good of a board. There are some really nice SM like the X8DTL-i for $300 and have almost identical features.
Get cheaper Xeons
Cheaper case - Norco 20 bay hotswap is $300 vs SM $2500
 
Could get one of the Supermicro cases with a built in expander. Won't have to deal with a ton of cables then. Also, the one without the expander is only $1400, not $2500.
 
Yeah, take a look at the Supermicro CSE-847E1-R1400UB which should be around $1700
Also, get the 1680i and forget the external cables for hooking up internal drives. Get a new generation raid card with internal and external connectors when you need to expand beyond 36 drives

Depending on how much processing power you need you can save money there too.
 
nitrobass24 said:
24x 2TB HDDs - $4080
Click to expand...

Only thing here is that you may not want to buy 48TB worth of drives. Odds are you will be running at least raid 6 and with 25+ drives, hot spares start making sense. In essense, you'll probably buy extra drives if you want that capacity for the sake of redundancy. This also means that you need to have enough ports and hotswap bays to accomodate those extra drives.
 
pjkenned said:
Only thing here is that you may not want to buy 48TB worth of drives. Odds are you will be running at least raid 6 and with 25+ drives, hot spares start making sense. In essense, you'll probably buy extra drives if you want that capacity for the sake of redundancy. This also means that you need to have enough ports and hotswap bays to accomodate those extra drives.
Click to expand...
This, with that many drives, you significantly increase your odds of having a bad drive somewhere at some point in time.
 
tgrimley said:
Is it possible to do this with $10k?

I need about 24TB for storage of recorded data, and about another 24TB to play with the data. I have used linux software raid for a number of years. Hardware raid sounds more attractive in this build, but I don't think it could be in budget.

Anyone have any suggested builds?

thanks in advance!
Click to expand...

Get the lianli A77 case and 4 supermicro 5-in-3 hotswap module. It will give you space for 23 drives (or 46TB, not exactly 48TB, but very close)

Lian Li A77

supermicro 5-in-3 hotswap module


The lianli A77 has 3 extra drive bay inside, so the total is 23 drives.
 
Thanks guys.

I'm pretty much guaranteed to have a bad drive at some point. I am wondering if a 24TB RAID6 array + 1 hot swap (15 drives) should be enough. So at least 30 drives in total.

Would I be foolish to attempt software raid at this array size and price point? I am worried about compatible drives being expensive..
 
tgrimley said:
Thanks guys.

I'm pretty much guaranteed to have a bad drive at some point. I am wondering if a 24TB RAID6 array + 1 hot swap (15 drives) should be enough. So at least 30 drives in total.

Would I be foolish to attempt software raid at this array size and price point? I am worried about compatible drives being expensive..
Click to expand...

You don't give an indication of the throughput you need. Nor an indication of the file sizes involved. It is really hard to generate 24TB of data and write it to disk. Manipulating 24TB of data is not going to happen on the recommended hardware. You don't have the ability to move the data to and from disk fast enough to get any work done. What amount of data are you really manipulationg?

Your disk costs are fairly modest for the value of the data. Given the probability of disk failure and the difficulty of rebuilding large arrays of data, I would get rid of the raid and simply keep a back up copy (mirror) of each hard drive. No need to rebuild an array, just swap hard drives.
 
Okay, some background.

We've got some uncompressed binary (compressible, but possibly too much overhead) data files about 1GB in size. Need to process and combine files to create another 24TB. Right now we're disk limited at ~120MB/s with an i5.

Server has two main functions:

1) store original untouched data set ~24TB
2) process and create new data set ~24TB

For 1 we need security, for 2 we need throughput. How much.. however much we can afford :)
 
Sequential reads with one of those Areca cards should be upwards of 800mb/s.
 
First off you say you only get 120MB/s with your i5.....So I am assuming you have a software RAID. It may be worth your time to get an Areca and just use what you have with hardware raid.

But if you want a new build
You could get two or three of the Areca's to spread the IO and bandwith, and create multiple Raid sets and do RAID 60

Do like 6x 6disk RAIDsets and then create a RAID60 Volume. This way each raidset can sustain a 2 drive loss at any given time and still be running smooth, with the potential to lose up to 12 drives at one time(2 for each raid set) and still be up and running.

Requires more hardware and will be more expensive but will be wicked fast, and have much higher availability.
 
"Need to process and combine files to create another 24TB. Right now we're disk limited at ~120MB/s with an i5."

120MB/s copies a 1GB file in 8-10 seconds. There is very little processing that you can do to a file in that much time. RAID will give you even less time for processing.

I think you are trying to solve the wrong problem.
 
GeorgeHR said:
"Need to process and combine files to create another 24TB. Right now we're disk limited at ~120MB/s with an i5."

120MB/s copies a 1GB file in 8-10 seconds. There is very little processing that you can do to a file in that much time. RAID will give you even less time for processing.

I think you are trying to solve the wrong problem.
Click to expand...

It does sound like he is CPU limited with the current setup, but if he uses Hardware RAID it would allow the CPU to perform the task of processing the data files.
Also this is obviously business related if they are will to drop 10k on a new system for it. Which i why i suggested Dual Nehalems.

All that aside the problem is not sequential speed its IO because right now all his CPU cycles are being taken up processing parity calculations instead processing the actual file.

Yea he may be getting 120 in a bench but when you are reading and writing simultaneously from the same raidset you will get about a 1/3 of that in throughput because when you are performing multiple reads and writes your data access is not longer sequential and has become random. Couple that with software RAID and your computer will be dragging ass.

Hardware RAID alone would help the situation a lot just by having something other than CPU to calculate parity and let the CPU do the real work at hand.
Add some more spindles and do a nested array and your IOPS will be much much higher than with a software RAID and you will be able to process these items much faster.
 
GeorgeHR said:
"Need to process and combine files to create another 24TB. Right now we're disk limited at ~120MB/s with an i5."

120MB/s copies a 1GB file in 8-10 seconds. There is very little processing that you can do to a file in that much time. RAID will give you even less time for processing.

I think you are trying to solve the wrong problem.
Click to expand...

Current set up is reading from one disk and writing to another, ~100-120MB/s on each disk, 40% CPU usage.

It's not my code processing the data, so I can't speak to exactly where the bottleneck is, but just report what I've seen monitoring a test set of 100GB data. My impression is that the processing is merely some interpolation and array reordering at this stage.

I appreciate the continued discussion :)

BTW, anyone have a recent areca 2tb hard drive compatibility list? the ones I can find are 6+ months old.
 
tgrimley said:
BTW, anyone have a recent areca 2tb hard drive compatibility list? the ones I can find are 6+ months old.
Click to expand...

HCL's are always lagging behind real world, especially in Areca's case. Hitachi 2Tb are flawless on an Areca 1680 and in your situation I'd deem it your only choice in light of the issues with the other brands of desktop class 2Tb drives.

I was hoping some people would chime in and give you some good advice but I see its all over the road. First question: what's your timeframe for this? ASAP? 2 months? Reason I ask is there's a new generation of SAS-2 RAID cards emerging in a couple months. Right now you'll be limited to about 600-650MB/s which is a limitation of the STP (Serial Tunneling Protocol) when you're attaching more than 8 drives and using an expander. With SAS-2 that should double.

That said, even with a current gen SAS RAID card you'll have incredible bandwidth to shuffle terabytes of data around. I frequently transfer between 20 and 24 Tb between arrays overnight- usually done in about 12-14 hours. 48 Terabytes is easy, you can definitely do it for under $10k, maybe even less than $7k. Main cost will be harddisks @ $150 each. If you went name brand instead of DIY, you'd spend 10-20x the price and have mediocre components at that- most of the major vendors use LSI chips on their cards and expanders, and mark everything up to high heaven. With DIY you'll get amazingly good cost to performance ratio.
 
Last edited:
thanks a lot guys, especially the experience with the hitachi drives. will report back when I pick a configuration :)
 
What's the difference between the SuperChassis 847E1-R1400UB and SuperChassis 847E2-R1400UB? They have different back planes but I cannot figure out what the difference is.

looks like I will be going with
32 2TB Hitachi drives
SuperChassis 847E1-R1400UB or SuperChassis 847E2-R1400UB
suggested cpu, ram, mb
Areca 1680xi-16

last piece I think is the raid card. It's unclear to me how the sas backplanes on these work, so I don't know what I will need exactly port wise.. anyone want to educate me? :)
 
do you need 24G of ram?

you get a SAS card that uses a SAS connector to attach to the back plane, the SAS connectors i have split off into 4 SATA, so each SAS more or less can connect 4 drives.
 
The E1 models have a single port backplane. The E2 models have a dual port backplane. Since you are not using any SAS drives, it will make no difference (go with the cheaper one). As for the RAID card, you only need a single port (well, single multilane, aka 4 ports). The easiest way to explain the functionality is to think of a SAS expander as a USB hub. You run a cable from your RAID card to it and you get more ports. When it comes to selecting a model, if you want something with expandable memory, then get the 1680 or 1680ix-12. If you don't need more than 512mb of RAM, then get a 1680i or 1680ix-8. One last thing to keep in mind. If you get a full-height RAID card, you will then need to get one of the UIO Supermicro boards and the appropriate chassis model as the standard model only takes half-height cards.
 
Recommend you stay away from any Areca with "ix" in the name. That means it has an onboard expander, which may introduce problems since you'd be daisy chaining expanders unnecessarily, by connecting an internal port on say an Areca 1680ix-12, to the expander chip on the Supermicro backplane. It would probably work but its not ideal, and with that many drives and that much data you want the signal chain as simple as possible with.

I recommend getting either a Areca 1680i (2 x SFF-8087), 1680LP (1 x SFF-8087, 1 x SFF-8088) or 1680 (2 x SFF-8087, 1 x SFF-8088). None of those have an onboard expander.
 
MrGuvernment said:
you get a SAS card that uses a SAS connector to attach to the back plane, the SAS connectors i have split off into 4 SATA, so each SAS more or less can connect 4 drives.
Click to expand...

This doesn't apply to the Supermicro chassis he's interested in, MrG. You're talking from the standpoint of a Norco 4220 or similar which has a simple backplane that just breaks the SFF-8087 connector out into 4 x SATA. The Supermicro chassis he's talking about has an expander chip on the backplane, which means he only needs a single SFF-8087 cable coming from the RAID card.

@tgrimley: If you ever wanted to expand beyond 36 drives, let's say for backup purposes, you could add another chassis without having to add a whole new motherboard to the second chassis- all you'd need would be a $30 Supermicro ATX power board (part# CSE-PTJBOD-CB1) to power on the system, 1 x SFF-8088 to SFF-8088 cable to connect the external port on the Areca card to the second chassis, and 1 x Supermicro CBL-0167L cable to connect the backplane of the second chassis to the cable coming from the first chassis.
 
Last edited:
ordered! thanks for all the help! will post results when parts stream in :)

@odditory: Went with the 1680IX-12 before I saw your message! Hopefully issues will be minimal.. About the second chassis, I was reading the docs and glad to see I understood correctly how you'd go about adding it. Now the waiting game..
 
Last edited:
odditory good to know! i will be ordering a chassis similar and did not know that!
 
tgrimley said:
ordered! thanks for all the help! will post results when parts stream in :)

@odditory: Went with the 1680IX-12 before I saw your message! Hopefully issues will be minimal.. About the second chassis, I was reading the docs and glad to see I understood correctly how you'd go about adding it. Now the waiting game..
Click to expand...

I would strongly urge you not to run the 1680ix-12 as long as you're going through so much time and expense. Why do everything else right but not get the right card from the beginning? I'll repeat, the 1680ix-12 has an onboard expander, which the Supermicro expander chip on the backplane would have to daisy chain through. The onboard expander on the Areca has presented some problems in my testing, and its not ideal to mix and match different SAS expander chips. You're adding an extra layer of complexity for no reason other than you already bought it and are hoping for the best?

A raid card driving driving THAT many harddisks is arguably the most critical component in the whole system, so I wouldn't leave this up to chance. If you end up having problems and ignore this advice, you are forfeiting your right to complain about it here. :)
 
Last edited:
odditory,

okay, I called and begged to cancel that part of the order and replaced it with a 1680lp. So now you know where I will come to bitch :)
 
good man. and please snap some photos of the build as it progresses, and post them in this thread. since you'd be the first person (i'm aware of) using one of those 36-Bay Supermicro's on this forum, I'm sure a lot of people would be interested in seeing it. That's all we ask in exchange for providing better real world advice on your build than any vendor or reseller ever could.

i am curious about one thing with that case - since it actually has two backplanes (a 4U in the front and a 2U in the back for each of the two drive cages) and therefore probably two expander chips, i'm assuming at least the 4U backplane in the front will offer a pass-through port to cascade the second backplane, and hopefully Supermicro went as far as pre-installing the SFF-8087 cable between the two backplanes. Since the Areca 1680LP only has one internal SFF-8087 connector to drive both backplanes, they'll need to cascade somehow.

I'm sure SM assumed a cascading scenario because everything they build is well thought out, I'm just curious how they implemented it, so let us know when you get the case!
 
Last edited:
From the parts list it looks like the cascading cable is an optional add-on.
 
Blue Fox said:
I believe that part is for cascading cases, not the internal backplanes.
Click to expand...

That was my impression too. I cannot understand in the chassis manual how the two are connected, but I will definitely take pictures and document issues when it arrives. hopefully it will help someone else make the decision
 
See Appendix E: SAS-826EL Backplane Specifications, E 9

The part list has both the internal cascading cables and the parts for externally cascading as well.

odditory has it right I think. According to the documentation, both the dual port and regular backplanes have the standard cascading connectors.

So internally, you would have to use SFF-8087<->SFF-8087 to cascade the backplanes. I wonder if SM includes those.

That would be these:
CBL-0108L-02 - SFF 8087 to SFF 8087 Internal Backplane Cable, 39cm
CBL-0281L - SFF 8087 to SFF 8087 Internal Backplane Cable, 75cm

You can continue to cascade these externally to another chassis.
CBL-0167L
CBL-0168L
I guess you'd best describe them as a SFF-8087 bracket connector that you connect from one of the backplanes. Then CBL-0166L connects this bracket connector to another chassis with another bracket connector.

Port wise, this chassis is actually cheaper then expected on a per port basis, given that the cascading expander's are figured into the price. I
 
Everything is in but the chassis.. wait wait wait. It's a damn pretty motherboard.. :)
 
Any update on the Supermicro 847E1-R1400UB build with the Areca 1680LP controller? I'm eager to hear how well the Areca 1680LP works with Supermicro's EL1 SAS expander.

I'm trying to decide whether to buy an 847A an use the HP SAS Expander, or to get the 847E1. I'd rather go with the E1, but I've seen reports here that the Areca works quite well with the HP SAS Expander, but I have yet to see anyone reporting smooth operation of the Areca with the Supermicro EL1 SAS Expander.
 
Unfortunately I am still waiting on the case to ship (backordered...) Will report back when it arrives, and take pictures with all the toys together. I can report that the Areca card is a dream thought :)
 
Hello everyone!

I'm very new to this forum, but after i found it i just can't stop reading.

I will post here my idea since OP had the same issue with choosing a config for his 48+ tb system.

Little background.

I work for a TV station as an IT guy, and now my mission is to build a system with as much TBs as possible in preferably 4U chassis.

So i googled my self one of SM 36 drives chassis. Also thought it would be nice to have some Adaptec 5Z series HBA. And of course 2TBs hard drives.

Now with this in my mind and with no hands-on experience spending company's 10k+ budget is kinda irresponsible act.

Now i found you guys.

So here is what i understand and please correct me if i'm wrong.

1. Buy SM x36 drives chassis with SAS expander on the backplane. E1 or E2 versions.
2. Buy mb cpu ram
3. Buy 36 Hitachi 2TB drives.
4. Buy Adaptec RAID 5445Z , 4int+4ext ports. Will that be enough to connect all 36 drives considering that there are expanders on the backplane? (external port is good for may be future expansion)
5. Config the RAID60 using 32 drives and other 4 use as 2 hot spares per each RAID6. Total space 56TB.

Or......

Right now i have up and running NAS based on SM 16 drive chassis, with Areca 1261ML in RAID6, Tyan mb.

So what left to do is:

1. Buy Adaptec RAID 5445Z. Install it in existing system.
2. Buy SM 36 drive chassis, buy HDDs, no mb+cpu+ram, and just connect it externally.

I guess this is cheaper than the first configuration.

What do you think?

Any optimization and tuning suggestions are welcome.

Thank you!
 
You must log in or register to reply here.
Back
Top