Need some info (mass storage experts)

Sexual Chocolate

Limp Gawd
Joined
Feb 25, 2003
Messages
237
Ive got 2 projects at work that Ive been tasked with tracking down some mega storage requirements.

Project 1:
1-2 petabytes of data recorded every 7 days, needs to be stored for 90 days so 12-24pb total.

Project 2:
20-30 petabytes of data recorded every 7 days, needs to be stored for 90 days so 240-360pb total.

Both projects require the drives to be removable and running some sort of redundancy. Ive tried the big dogs (EMC, NetApp & HP) and it doesnt seem like they really have a solution.

Im just looking for some pointers on who else I can look to that can handle a project of this enormity.

Thanks
 
Holy geez...

And here I thought the 20TB ZFS system I have, or the 100TB systems people build and post here was big.

I can't imagine the physical space requirements necessary.

Seems like you would have to do something custom to even get close.
 
Surprised HP didn't mention SFS. It should fit at least the smaller project, but it has more than enough speed to handle the larger one as well (since you need at least 50gb/s for that one).
 
Time to contact the LHC

Good luck man

Sounds like Windows Home Server is the solution
 
Questions:
1. What do the transfers look like? E.g. is project 1 sustained +/- 10% 24x7 or is this twelve hours a day at high burn and ten off.
2. What level of redudnancy do you want.
3. Do you have the space/ people for this?
4. Do you have the budget for this? Odds are you are not going to buy 386PB total and be done.
5. What will storage growth like (if applicable)?
6. What protocols do you need to use?
7. What systems will be putting out the source data?

If you would rather discuss over e-mail (I am guessing this is at least semi-sensitive), please PM me with contact information.
 
Pikenned, thanks for the reply. I'm not able to give a lot of specifics on this project but I will try and give you what I can.

1. The data will be coming in over a 40Gigabit pipe, somewhat steady for the entire run (7 days).

2. We would need some very solid redundancy. Single failure is not good enough.

3. Part of figuring this out is determining how much space and people we will need. We are limited in the space size sort of... meaning I have an 8x20 space to work in but I can have multiples of those 8x20 spaces as needed. People will not be an issue.

4. Budget is not an issue unless the solution is going to require something yet developed that would require us to help pay for development costs. (There is a SSD solution where we can fit 1PB in a harding case, but its not fully developed and they want a mill or more to finish it quickly).

5. Growth not required, after 90days data will be destroyed and replaced by the next data set.

6. Im not able to elaborate on this

7. Im not able to elaborate on this other than to say the source data is imagery and it will already be somewhat compressed.

It is somewhat sensitive, but I'll keep the response here in case anyone else might have any ideas and then we can move the conversation elsewhere if needed.

Thanks.


Questions:
1. What do the transfers look like? E.g. is project 1 sustained +/- 10% 24x7 or is this twelve hours a day at high burn and ten off.
2. What level of redudnancy do you want.
3. Do you have the space/ people for this?
4. Do you have the budget for this? Odds are you are not going to buy 386PB total and be done.
5. What will storage growth like (if applicable)?
6. What protocols do you need to use?
7. What systems will be putting out the source data?

If you would rather discuss over e-mail (I am guessing this is at least semi-sensitive), please PM me with contact information.
 
Sounds like you need to talk to Cray/SGI/IBM :eek:

They have proprietary high speed interconnects, etc
 
Your problem is removable. You can do it but look at not removing drives but removing arrays.
 
You're going to need some serious hardware to fit these specs. I've done some high level enterprise solutions around this idea. Eg: http://www.hds.com/products/storage-systems/hitachi-virtual-storage-platform.html?_p=v

The Hitacht VSP allows you to throw a LOT of storage behind a single (or multiple) high speed head ends. It will allow you to grade your data from high performance, mid performance and archive, as well as do all the nice dedupe and compression stuff (although you've already mentioned you've got your data fairly well compressed).

If you want more useage cases or discuss privately, let me know.

* 192 GB/sec aggregate internal bandwidth
* Up to 255PB of total storage capacity under management, including internal storage up to 2,048 SAS 2.5” drives (1.2PB of storage) or 1,280 SATA 3.5” drives (2.5PB)
* Up to 256 flash drives supported for exceptional performance
* Unmatched multiplatform connectivity: up to 192 Fibre Channel 8 Gbits host ports, 192 IBM® FICON® 8 Gbits host ports, 96 Fibre Channel over Ethernet (FCoE) host 10 Gbits port and network-attached storage (NAS)
 
I'd definitely like more info on this. It sounds like it would work for the smaller of the 2 projects which we estimate at a maximum of 35PB, however its a ways off for the larger.

Send me a PM with some contact info.

Thanks



You're going to need some serious hardware to fit these specs. I've done some high level enterprise solutions around this idea. Eg: http://www.hds.com/products/storage-systems/hitachi-virtual-storage-platform.html?_p=v

The Hitacht VSP allows you to throw a LOT of storage behind a single (or multiple) high speed head ends. It will allow you to grade your data from high performance, mid performance and archive, as well as do all the nice dedupe and compression stuff (although you've already mentioned you've got your data fairly well compressed).

If you want more useage cases or discuss privately, let me know.

* 192 GB/sec aggregate internal bandwidth
* Up to 255PB of total storage capacity under management, including internal storage up to 2,048 SAS 2.5” drives (1.2PB of storage) or 1,280 SATA 3.5” drives (2.5PB)
* Up to 256 flash drives supported for exceptional performance
* Unmatched multiplatform connectivity: up to 192 Fibre Channel 8 Gbits host ports, 192 IBM® FICON® 8 Gbits host ports, 96 Fibre Channel over Ethernet (FCoE) host 10 Gbits port and network-attached storage (NAS)
 
Have you looked into Lustre as the filesystem? I remember from the conference a few years ago (2 or 3) that all the big labs used Lustre....not so great on Iops as there is an "rpc-in-flight" limit of about 15k or so IIRC.
 
Usually people who store that much data know other people who store that much data. Those people are willing to share their construction details.

360PB of data really needs more information to determine what type of storage system is needed. I expect the 24PT data set would allow for testing of the large system model.

---

I suppose that if you are storing 2PT data sets, someone is doing some processing on the data sets.

---

Removable drives. I guess mounting 10,000 (20,000 for redundancy) drives in a van makes them removable. Maybe 1000 in each of 10 vans.

You have $1 million in hard drives and each one needs to be formatted and verified. Law of big numbers is against you.

As I have said. There are people who have set up systems this large. Go talk to them. Buy them lunch. Give them lots of money.
 
As I have said. There are people who have set up systems this large. Go talk to them. Buy them lunch. Give them lots of money.

seriously. anyone with this kind of experience isn't going to be offering up solutions to large corporate/enterprise storage challenges for free on a hobbyist/enthusiast storage forum, at least anyone worth their salt.

a lot of "help me solve my corporate I.T. problems" threads on here lately.
 
Thanks for the feedback guys. Just to clarify a few things though I wasnt looking for a solution, I clearly asked for some "pointers on where to look". When I was asked for some additional information I provided it.

I disagree that you wont find a lot of experts floating around these forums, Ive been reading posts here for years and almost always find expert help. While this territory is new to me, I actually did get some very good feedback thus far that has helped me out.

The Hitachi Virtual Storage response was great and Im waiting for a call back from them now. Ive also already spoke to IBM who not only offers the XIV solution as mentioned above but other IBM SONAS options that might work (The DS5300 w/ EXP5020 expansions).

Ive also had a NetApps guy message me who might have something that I overlooked originally and who knew that SGI had some really good options??

Oh and regarding the comment about buying people lunch, its seems that the storage companies are looking to buy me lunch today :)

Thanks
 
Back
Top