Methods To Combine Storage

[GreenLED]

I have tried throughout today to get an idea of how I should go about building a scaleable storage server that I can keep expanding using the same methods. I have taken a look at the Showoff Thread where people have built out large amounts of storage - quite impressive I might add.

I feel like I'm not getting anywhere in terms of a plan for how I should build this storage server.

What I really don't understand, and maybe you all can clarify this for me, is - what methods are people using to combine large amounts of storage together. I see on a lot of those posts in that showoff thread people have built out servers - complete with motherboards and all the normal components of a PC and they have taken multiples of these and just have them put together somehow. Am I missijng something? I would think the more efficient way to do it would be to build one box that is the controller, with one motherboard, ram, etc. Then build out "expansion boxes" that basically are just a power supply, hard drives, some sort of cooling mechanism and (of course) a way to bind that data to the rest of the hardware already on the main server.

I feel like I'm talking about some super-expensive solution, and I very well could be. The point is, are there only two methods to pool together drives? (i.e. port multipers and SAS expanders)

Please enlighten me.

Oh, and what in the heck is a backplane?

 

[drescherjm]

Do you really need all of your storage to be combined in one place? I mean at work where I have around 70 TB of mostly raid6 storage spread out on 6 linux servers it is not really combined. I do not have a single 70TB partition.. The 70TB is not on a single filesystem. The raids are not pooled together at all. Although the windows users do see a single share J: that contains links to most of this storage. The links are mounted as folders on samba using msdfs to look like a tree mostly part of that single share that is mounted as j:

Oh, and what in the heck is a backplane?

As far as storage it can be a card that 1 or more hard drive plugs into for hot swap capability.

 

[Dangman]

I see on a lot of those posts in that showoff thread people have built out servers - complete with motherboards and all the normal components of a PC and they have taken multiples of these and just have them put together somehow.

Could you please provide examples of that from that thread? That would help us get a clearer idea of what you're looking at and see whether or not you're misinterprating something.

Though off the top of my head, more than likely they're using each server for a different purpose rather than a common storage pool. They might even be doing some clustering related work. Or, as drescherjm kinda pointed out, not everyone wants all of their storage combined in one place.

Am I missijng something? I would think the more efficient way to do it would be to build one box that is the controller, with one motherboard, ram, etc. Then build out "expansion boxes" that basically are just a power supply, hard drives, some sort of cooling mechanism and (of course) a way to bind that data to the rest of the hardware already on the main server.

That certainly is a good route:
http://www.servethehome.com/category/client-tips-and-tricks/the-big-whs-30-drive-whs/

But again, without know what exactly what those other people were doing with their server setups, it's actually hard to say whether or not that "efficient" way is actually the best option.

I feel like I'm talking about some super-expensive solution, and I very well could be. The point is, are there only two methods to pool together drives? (i.e. port multipers and SAS expanders)

Well those are the two more common and easiest methods to hook as many hard drives as possible to a server. Other methods may involve iSCSI, Fibre, Infiniband, etc. There's also ways to pool drives together through the network as drescherjm has somewhat illustrated.

Oh, and what in the heck is a backplane?

Wikipedia is your friend (HINT: look for "Storage"):
http://en.wikipedia.org/wiki/Backplane

 

[drescherjm]

Or, as drescherjm kinda pointed out, not everyone wants all of their storage combined in one place.

One advantage that I can think of (at 2AM - struggling to keep my eyes open) is that I can reboot one or more of the 6 servers and still have the j: drive available. Obviously the storage on the server that goes down goes away but its not like at all times every part of our data storage is needed. I check if anyone is accessing the server that I need to reboot, send out an email explaining the brief downtime then do my maintenance.

A second advantage to not having everything in a single raid is that raids and hardware can fail. I do not want to have all of my storage on a single raid even if it was a zfs zpool with multiple raidz3 vdevs and even though I do backup using a 2 drive tape autoloader.

 

[Dangman]

One advantage that I can think of (at 2AM - struggling to keep my eyes open) is that I can reboot one or more of the 6 servers and still have the j: drive available. Obviously the storage on the server that goes down goes away but its not like at all times every part of our data storage is needed. I check if anyone is accessing the server that I need to reboot, send out an email explaining the brief downtime then do my maintenance.

A second advantage to not having everything in a single raid is that raids and hardware can fail. I do not want to have all of my storage on a single raid even if it was a zfs zpool with multiple raidz3 vdevs and even though I do backup using a 2 drive tape autoloader.

Another advatange I can think of (It's also well past my bedtime as well) is that you might have a lot of extra CPU, mobos, and RAM lying around and want to put it to some use. Or they can't simply afford the somewhat high costs for an extra SAS expander and the necessary equipment to keep that SAS expander connected and powered.

 

[brutalizer]

You can connect an external JBOD (DAS) case to your HBA card. HBA is just a dumb raid card, which connects your pc to the external disk case, via a single SFF8088 cable. I have this external case:
http://www.ebay.co.uk/itm/NEW-Super...535766?pt=COMP_EN_Servers&hash=item3375bb3756

connected to my PC via a single sff8088 cable. You need a HBA for this, typically an IBM M1015 flashed to IT mode. Or, you can use a DELL 6GBps HBA card which is also cheap (it has external connectors), whereas IBM M1015 has internal connectors (so you need an adapter which transforms internal connectors to external)

Because my JBOD case has SAS expanders, I can use a single minisas sff8088 cable. Each SAS cable connects normally four disks to each miniSAS port in the HBA.

 

[GreenLED]

Do you really need all of your storage to be combined in one place? I mean at work where I have around 70 TB of mostly raid6 storage spread out on 6 linux servers it is not really combined. I do not have a single 70TB partition.. The 70TB is not on a single filesystem. The raids are not pooled together at all. Although the windows users do see a single share J: that contains links to most of this storage. The links are mounted as folders on samba using msdfs to look like a tree mostly part of that single share that is mounted as j:



As far as storage it can be a card that 1 or more hard drive plugs into for hot swap capability.

Ok, so if this linux app is binding together the storage into a single "J" drive, what is managing spillover? What decides where a file is placed physically (i.e. what server does this file go to?), is it split across storage? I just don't understand how this is working. Let's assume I have a user how is copying a file that will be bigger than his "cubical" of space I have setup for him/her, what happens? Is this linux daemon somehow managing things? I just am not getting this properly in my head.

 

[GreenLED]

There's a lot to respond to here guys - a lot for me to think about. I'm going through and trying to parse everything. Give me some time to answer.

 

[GreenLED]

You can connect an external JBOD (DAS) case to your HBA card. HBA is just a dumb raid card, which connects your pc to the external disk case, via a single SFF8088 cable. I have this external case:
http://www.ebay.co.uk/itm/NEW-Super...535766?pt=COMP_EN_Servers&hash=item3375bb3756

connected to my PC via a single sff8088 cable. You need a HBA for this, typically an IBM M1015 flashed to IT mode. Or, you can use a DELL 6GBps HBA card which is also cheap (it has external connectors), whereas IBM M1015 has internal connectors (so you need an adapter which transforms internal connectors to external)

Because my JBOD case has SAS expanders, I can use a single minisas sff8088 cable. Each SAS cable connects normally four disks to each miniSAS port in the HBA.

There's a lot here for me to ingest and understand, but I really like this method A LOT.

"A host bus adapter (HBA) is a circuit board and/or integrated circuit adapter that provides input/output (I/O) processing and physical connectivity between a server and a storage device." (from Google).

So, this HBA is something plugs into a PCIe slot I'm assuming correct?

More importantly -- instead of buying a FULL SERVER chassis, can I buy a case that includes room for JUST HDDs, power supply and cooling fans? Does this type of chassis come with the connector that will connect to all my HDDs or is that something separate I have to purchase?

I really like this method.

 

[drescherjm]

Ok, so if this linux app is binding together the storage into a single "J" drive, what is managing spillover? What decides where a file is placed physically (i.e. what server does this file go to?), is it split across storage? I just don't understand how this is working. Let's assume I have a user how is copying a file that will be bigger than his "cubical" of space I have setup for him/her, what happens? Is this linux daemon somehow managing things? I just am not getting this properly in my head.

dfs does not work like that. You get a tree of folders where any of the subfolders can be redirected to any other cifs server share (windows or samba) on the network. In your example when a user writes to a folder and the filesystem containing that folder is full the write will not succeed. It does not try to place the file somewhere else or pool where the file goes.

So, this HBA is something plugs into a PCIe slot I'm assuming correct?

Yes a HBA is a PCIe card.

 

[brutalizer]

There's a lot here for me to ingest and understand, but I really like this method A LOT.

"A host bus adapter (HBA) is a circuit board and/or integrated circuit adapter that provides input/output (I/O) processing and physical connectivity between a server and a storage device." (from Google).

So, this HBA is something plugs into a PCIe slot I'm assuming correct?

The HBA can be a hardware raid card which has been flashed to IT-mode, that is, all raid functionality has been flashed away. The raid card is empty now, so it is only used to connect external cables to it. It can look like this:
http://forums.overclockers.com.au/showthread.php?t=1045376

More importantly -- instead of buying a FULL SERVER chassis, can I buy a case that includes room for JUST HDDs, power supply and cooling fans? Does this type of chassis come with the connector that will connect to all my HDDs or is that something separate I have to purchase?

This case I showed holds 45 disks. It has one (or two cables if you need more perfomance) cable, that can be 10 or 20m long. It is called sff8088. It is connected to your pc, via a HBA (a dumb card). It has miniSAS connectors, so the only thing you need to do, is to populate the case with 45 disks, and connect it to your PC (via an sff8088 cable to the HBA). The case is empty, it holds only disks - that is what JBOD is short for: a case that only holds disks. Or it can also be called DAS. Just google for JBOD or DAS cases, all of them holds disks. It is empty except a connector and a backplane. You dont build a PC inside a JBOD case - they are supposed to only hold disks.

So you just insert your disks into the external JBOD case, and connect it to your PC. And then the PC will believe it has 45 disks inside its chassis, when in fact, all 45 disks sit in another external case. This will give you 2Gbyte/sec bandwidth. If you use two cables, you will get close to 4GB/sec read write speed.

The cheap DELL or IBM HBA card holds two sff8088 ports. This means that you can connect two such cases to one HBA, so you can have 90 disks. If you need more disks, you just add another HBA. There exists expensive HBA card, which has four sff 8088 ports.

 

[GreenLED]

The HBA can be a hardware raid card which has been flashed to IT-mode, that is, all raid functionality has been flashed away. The raid card is empty now, so it is only used to connect external cables to it. It can look like this:
http://forums.overclockers.com.au/showthread.php?t=1045376


This case I showed holds 45 disks. It has one (or two cables if you need more perfomance) cable, that can be 10 or 20m long. It is called sff8088. It is connected to your pc, via a HBA (a dumb card). It has miniSAS connectors, so the only thing you need to do, is to populate the case with 45 disks, and connect it to your PC (via an sff8088 cable to the HBA). The case is empty, it holds only disks - that is what JBOD is short for: a case that only holds disks. Or it can also be called DAS. Just google for JBOD or DAS cases, all of them holds disks. It is empty except a connector and a backplane. You dont build a PC inside a JBOD case - they are supposed to only hold disks.

So you just insert your disks into the external JBOD case, and connect it to your PC. And then the PC will believe it has 45 disks inside its chassis, when in fact, all 45 disks sit in another external case. This will give you 2Gbyte/sec bandwidth. If you use two cables, you will get close to 4GB/sec read write speed.

The cheap DELL or IBM HBA card holds two sff8088 ports. This means that you can connect two such cases to one HBA, so you can have 90 disks. If you need more disks, you just add another HBA. There exists expensive HBA card, which has four sff 8088 ports.

I feel like the last two days haven't been wasted. Thank you for that response and for stepping through how this all works. I think I'm going to use this method. Is it possible to upgrade these cards later on to increase the GB/s?

 

[GreenLED]

Well those are the two more common and easiest methods to hook as many hard drives as possible to a server. Other methods may involve iSCSI, Fibre, Infiniband, etc. There's also ways to pool drives together through the network as drescherjm has somewhat illustrated.

Before I make a final decision on this, I want to also explore the options you talked about here. I imagine that the cost of using fiber would be mostly in the cards that adapt things so they can talk to each other. The fiber cables themselves would cost something, but they'll be short runs, so I wouldn't expect they would be too expensive.

The method I looked at in the previous post seems good, but it has a few limitations. First, if I can only get 4GB/s, that is 2GB loss from being able to saturate a 6GB/s drive. Now, in reality, not a big deal for now. Also, at some point, I will run out of PCIe slots. I know I'm in an overkill mindset, but I have to be this way to understand what the potential is here. Anyhow, I guess what my question is, what do big server farms do to cluster things together? It feels like they have some sort of way to daisy chain every storage pool they have into one -- or maybe that's my imagination running wild! Either way, I'd like to break the 6GB/s bottleneck at least. I thought I saw a post (may have even been this one, I've looked at so much stuff in the last 24 hours) that described a system able to break 24GB/s.

Thank you for putting up with my constant questions. I want to learn and to do that, I have to be that guy in class that makes everyone stay late .

 

[drescherjm]

if I can only get 4GB/s, that is 2GB loss from being able to saturate a 6GB/s drive

There is no such thing as a sata drive that can deliver 6 Gigabytes per second. The fastest SATA drives of today top out near 200 megabytes / second. brutalizer was talking in gigabytes not gigabits.

 

[GreenLED]

You are correct. I knew this. I'm a dummy . If you're combining drives though, and you have striping going on (as well as redundancy), you would break that 200 MB/s, no?

 

[drescherjm]

In sequential transfers yes very easy to break 200MB/s. I have seen around 1GB /s sequential reads with 10 to 12 drives in raid6.

 

[brutalizer]

You can cluster using different solutions. If you are going to use some OpenSolaris derivative with ZFS, you can use RSF-1 clustering software, which is expensive. There are open source versions, google on my name and RSF-1, for a link with open source clustering software.

Regarding read/write speed, you can create a zpool of SSD disks if you really need high performance. One guy in www.servethehome.com forum, created a server with two external JBOD cases, with 48 high performing SSDs for dataware house use - reaching 18GB/sec
http://forums.servethehome.com/diy-...warehouse-database-storage-servers-cheap.html

 

[GreenLED]

"You gonna make me loose my mind!"

That's crazy bro. Really fast. I'll take a look at that.

Also, thanks for letting me know about the clustering, I REALLY wanted someone to expand on that. I' will take a look at that as well.

 

[drescherjm]

On the linux side I know that lustre is now used as clustering filesystem on top of zfs. That was actually the main reason for the native zfsonlinux port to linux.

http://zfsonlinux.org/

 

[SirMaster]

What about distributed filesystems like Ceph. I bet some people are using that.

http://en.wikipedia.org/wiki/Ceph_(storage)

 

[SirMaster]

What's the technology called that converts the sff8088 into 45 SATA ports?

How can I just get that technology in my existing PC? If I have an IBM M1015 how can I connect say 12 SATA drives to it's 2 sff8088 ports in my existing case?

I'm aware of the sff8088 to 4 SATA cable, but how do I get more than 8 drives on a single IBM M1015 in an existing normal case?

 

[brutalizer]

You connect one sff8088 cable to 45 disks by SAS expanders. The JBOD has one or several SAS expanders. One sas expander, has several sff8088 connectors. Each sff8088 connector handles four SATA/SAS disks. So one sas expander, might have 6 sff8088 connectors = 6 x 4 = 24 SATA/SAS disks. So for that particular 45 disk JBOD, it has two sas expanders, and you need to connect them to each other.

JBOD has two SAS expanders, they are connected to each other (daisy chained). Then you can connect only one sff8088 cable from your IBM M1015 to the JBOD case.