Beginners guide to Raid cards / SAS expanders / Large storage systems?

smidley

Weaksauce
Joined
Mar 3, 2010
Messages
114
So, I'm going to be building out a new server over the next few months. I'm looking for maximum storage size. I'm looking at doing around 20x 2tb SATA drives. Because of this, I've been doing a lot of research on raid cards and expanders. However, I still have some confusion about some things, and it seems like the more I read, the less I know. :p And I'm willing to bet that there's more than a few other people out there in the same boat.

I was wondering if anyone would be willing to put together a little primer on how all of this works, and what kind of devices you would need to put a server like this together. Things would include: Power, backplanes, raid cards, expanders, etc. If we get a good thread with this kind of info in it, it would probably make a good sticky item in the forum. Thanks!
 
Last edited:
There are tons of threads about this. The big issue is how many drives you want to have, where the server is located, and what OS you want to run. Those three factors have huge implications on everything else to the point where you can't generalize beyond that.

Here's my blog with a lot of info you are probably looking for. http://www.servethehome.com

Anything specific you would like to see covered feel free to suggest an article here.
 
Well, I guess I'm talking about something that describes how everything is physically connected together. For instance, if I have a raid card, and I want 20 drives run off of it, how many expanders will I need? what types? How do the fan-out cables work and what are the different types? Things like that.
 
You probably will want to read up on SAS (Serial attached SCSI). Wikipedia has a decent page about it.

As far your questions go, the cables are directional, so you need to make sure you have the right ones. # of expanders would depend on the expander/backplanes you choose to go with.
 
Well, I guess I'm talking about something that describes how everything is physically connected together. For instance, if I have a raid card, and I want 20 drives run off of it, how many expanders will I need? what types? How do the fan-out cables work and what are the different types? Things like that.

If you are talking about a non windows based system you will not need or want any raid cards or expanders for 20 hard drives and could do the server for less than $1000 US total for a system that nets over 1GB /s reads and writes. This price would include a 20 or 24 bay hotswap case and everything but the hard drives. So you need to be very specific on what exactly you want.

You can do this with a combination of a motherboard that has 14 ports already and add a $100 to $200 8 port SAS card.
 
Last edited:
For my setup, I'll probably be running Server 2008 R2. I would like to know how the power supply would connect to a backplane of a case like the norco 4220. How is there enough power connectors in a single power supply to power 20 drives? Or do you have to use Y adapters? What would be a good Raid card to start with for this many drives? Do I need to worry about BBU? Raid cards normally only have a few ports on it, so what do I need to buy as far as expanders / cables go to connect up this many drives? Thanks for the help so far guys. I've read quite a bit about sas protocols, and a couple of the connectors, but some of it is still confusing to me.
 
For my setup, I'll probably be running Server 2008 R2.
That will most likely double the cost of the server but its good to go with what you know.

How is there enough power connectors in a single power supply to power 20 drives?

It looks like the 4220 takes care of that.

http://www.norcotek.com/RPC-4220.php


Or do you have to use Y adapters?

If the case did not already do that you could get a few of these.

http://www.newegg.com/Product/Product.aspx?Item=N82E16816133040

Just do not put them on the same chain..


Do I need to worry about BBU?
If you want good write performance and value your data you must get the BBU. My advice is check eBay. I have seen $15 BBU units.

Raid cards normally only have a few ports on it, so what do I need to buy as far as expanders / cables go to connect up this many drives?

You probably want to get a 8 port SAS/ SATA hardware raid card with a BBU and 1 or two expanders.

Edit: I take that back. I would probably get something like the following:

http://www.newegg.com/Product/Produ...6151073&cm_re=SAS_raid-_-16-151-073-_-Product

or this:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816151042
 
Last edited:
So, for the Norco 4220 backplane, you plugin 10x molex power connectors and it distributes it accordingly to the 20 drives?
 
So, for the Norco 4220 backplane, you plugin 10x molex power connectors and it distributes it accordingly to the 20 drives?

From the picture it appears that is what it does. I do not have this case so I am just looking at the picture..
 
A backplane typically takes a molex power connector and powers 4-5 drives. So in a 4220/4020, you would need 5 molex connectors. I used Y adapters with no problems, but this was with a single rail PSU. YMMV with PSUs with multiple rails. You don't need to connect all 10, just. It seems to be for redundant power supplies, but I haven't tested this.

On a 4220, you would need to connect 5 cables with target SFF-8087 connectors to your expander/HBA(s). The initiator/host end would be whatever connectors your expander/HBA(s) use (typically SFF-8087 as well).

What RAID card you buy depends on your budget. What SAS expander you buy depends on compatibility with aforementioned RAID card. You probably won't need more then one expander unless it has less then 4-5 SASx4 ports on it for some odd reason.
 
Thanks for all the excellent info guys! I feel like with all my reading and the posts here, I've almost doubled my understanding of this stuff today. :)
 
One more question: Can someone explain how expanders generally connect up to a raid card? And is there any benefit to buying a Raid card with more ports, rather than buying a raid card with limited ports and a less expensive expander to hook up to it? Thanks.
 
They generally take one 4 port SAS connection and divide it into many 4 port SAS connections.

Here is an example.

http://cgi.ebay.com/HP-24-Port-SAS-...ltDomain_0&hash=item5641a96fcd#ht_2545wt_1139

This takes 1 input 4 port mini SAS connector and from that you can connect up to 24 drives on the 6 output SAS connectors. Obviously they share the bandwith but since SAS is 3 Gbit /s full duplex a single mini sas has I believe 1200 MB/s bandwith.

And yes you can use these to reduce the cost.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
BTW some if not many of the larger port SAS cards are large port counts because they use an internal expander.
 
Thanks for the info. Right now I've got a Dell poweredge 2950 with a Perc 6/i Raid controller, and 6x 2tb drives in it. Unfortunately the perc controller has a different connector type and looks like it can only support (I think) up to 8 drives.
 
That Perc 6/i controller does support expanders. I am not sure about which ones are compatible. But this is discussed in this forum a lot being that Perc 6/i cards are sold on eBay for less than $200 US.
 
I had this same problem trying to understand raid cards, sas, etc. There are no really good definitive guides.
 
That's the wrong cable I believe.

Overview from manufacturer site says: 18" Internal Mini-SAS SFF-8087 to SFF-8484 controller to backplane cable.

You would need SFF-8484 controller to SFF-8087 backplane.
 
That's the wrong cable I believe.

Overview from manufacturer site says: 18" Internal Mini-SAS SFF-8087 to SFF-8484 controller to backplane cable.

You would need SFF-8484 controller to SFF-8087 backplane.

I didn't know cables were directional... is that right?
 
Yes, they are. Break out cables are named forward or reverse. Other SFF cables usually mention Host/Target/Device/Initiator/Backplane/Controller to distinguish between the ends.
 
Believe the new SuperMicro USAS2-L8e could connect alot more disks via the expander, more than the usual L8i versions.

Buying an extra controller is no option? For things like UnRAID/FlexRAID i can understand Expanders make some sense; those will act as JBOD if i understand correctly, and those only involve one disk in I/O; meaning expanders won't cost you any performance.

Still would be nice to compare the price of an extra controller versus expander setup. A controller without expander just works, but with expander there may be issues or compatibility bugs. If you don't really need an expander but a second controller would also be sufficient, then i personally would prefer that last option instead.
 
I believe the OP wants a raid 5/6 card since he is using windows. Otherwise HBAs would be a good option.
 
If he's going to use FlexRAID i believe he may still prefer a HBA instead, especially since this could mean no more disk dropouts due to UBER/bad sectors. For RAID5 on Windows he would have to use Hardware RAID + BBU for writeback + TLER-capable disks for a reliable storage setup, which is kind of expensive.
 
I am firmly of the opinion at this point that one should only run a controller in RAID mode when using some sort of software RAID in very limited instances. It is a bad idea to have an OS-RAID implementation, a hardware RAID implementation, and then the disk controller all trying to figure out what is going on when reading physical media.
 
I am firmly of the opinion at this point that one should only run a controller in RAID mode when using some sort of software RAID in very limited instances. It is a bad idea to have an OS-RAID implementation, a hardware RAID implementation, and then the disk controller all trying to figure out what is going on when reading physical media.

What are you talking about, My ZFS Raid 2Z + RAID10 HighPoint card + JBOD iSCSI box is running fine
I can't get it to boot though, I'll start another thread about it, I wonder if it's because it's all connected through USB -> eSATA -> SAS adapters....
 
Very helpful thread. Been looking to upgrade from a perc 5 raid 5 array to raid card/hp expander combo for a norco 4224 setup with 16 1TB's and 8 500GB's.
 
Yea, it would be nice to somehow recreate this thread, but more organized. Then we could make it a sticky. lol.
 
I kind of agree with smidley, but maybe we've reached the limits of what a discussion forum can achieve as opposed to an encyclopedia-like site.
All the useful information is there, but it's spread across dozens of threads, some of them (that I read fully at some stage) are just plain unreadable, like the HP expander thread of over 60 pages long.

There is really no guide on the web about how to setup a large storage, what the options are and how do they compare, the systematical inclusion of a backup system, how to best serve these files, etc.
Pjkenned's site and some others are most practical and informative, but they usually blog about one specific experience, they are not a guide.

I came to this specific Data Storage Systems to learn, and I learned a lot, but I would have learned much more and much faster from a technical storage guide than the painful process of reading thousands of unorganized posts, with only a minority being useful. Again, that's no discredit to the wonderful [H], it's just how discussion forums work.

I'd also love to see guides for needs such as 2-6 TB up to 20-100 TB of data (not the size of the whole storage) that link to wikipedia-style articles for all the reference data about specific matters such as OS RAID vs hardware RAID vs software RAID (that was mentioned earlier this week in another thread,) DAS vs NAS, JBOD vs RAID, summary practical information about specific models of enclosures, expanders, etc. (which probably wouldn't fit in wikipedia itself.)

Also with up-to-date information, not obsolete models or technologies such as SATA 3 Gb/s.
One article at a time...

The keyword here is "usable reference information" which cannot exist in a discussion thread, even if the same information is present somewhere. Most people don't want to memorize all that technical information or jargon, that's not their job or main concern, they have their own things to memorize, but they still want to quickly find answers to the questions they have, based on speed and budget requirements.
 
@Chimel what you're suggesting about writing comprehensive guides sounds like a fulltime job for someone, are you volunteering? :) Writing guides is a thankless job, on top of it technology is changing so fast that keeping it updated really ends up being the most time consuming part. On top of that there's always a lot of disagreement over the best approach to doing something - hardware raid, software raid, whatever - as soon as a "Guide" is written you've got ten people disagreeing with the information.

smidley's viewpoint in starting this thread - the idea of trying to consolidate information - isn't exactly a new one. for a long time myself and others have offered to organize this forum better with guides and stickies, but the owner couldn't be bothered and wasn't interested. case in point sub.mesa - hard to argue he shouldn't have some management ability in this forum with all the time and effort he's put in. as it stands now the only "modding" going on here is generally just spam-guarding. meaning good luck getting anything stickied. and without active, involved mods, even if you stickied the best guide in the world on say, "Guide To Imaging Software", it'd still be met with the same new ten threads a day asking "what's the best imaging software". guaranteed.

so [H] will continue to be a great forum with a lot of information but will remain hamstringed by its unmanaged nature. a lot of people have put in a lot of time here trying to provide ideas and help people out, but some of us are ultimately moving on elsewhere to where things can be managed and organized better.
 
Last edited:
a lot of people have put in a lot of time here trying to provide ideas and help people out, but some of us are ultimately moving on elsewhere to where there will be more active organization.

Any suggestions on where things are better?
 
@Chimel what you're suggesting about writing comprehensive guides sounds like a fulltime job for someone, are you volunteering? :)

@odditory Yeah, I hesitated a bit before posting, because I didn't want to give the impression that I'm asking anyone on [H] to comply with my request! ^-^
I plan to build a new server next year, so I'll be experimenting with some of these technologies then. I am willing, but I really have no expertise in storage and only my own limited experience. I mentioned "one article at a time" in the hope that it could be some collaborative project. It's too big and boring for just one person. Maybe we can give this assignment to some IT school? ^-^
 
Back
Top