Starting a data vaulting company

J

jdm71488

n00b
Joined
Sep 28, 2005
Messages
33
I have a friend who needs to store 2TB worth of data. It is all simple storage and will be somewhat of an archive that she can SFTP into at will. That is, of course, until I can get a more sophisticated Web-based interface or automated client-based solution in-place.

I've looked at Rackspace but don't have the revenue to justify paying $2000+/mn for hardware. I've looked at Mosso, but don't like that I cannot FTP data in and I don't care for "cloud" storage. I plan to buy my hardware and co-locate.

I am thinking of going with a light Dell PowerEdge server (1xCPU, 1-2GB ram, 2x80GB HDDs with RAID 1) using CentOS 5. Attached will be a Dell MD3000 to provide enough for the time being along with the capability to scale up to ~45TB.

My concern is that although I am RAIDed, I don't have a true redundant backup solution in-place. I don't want to invest in becoming a "data vaulting" company, even a small one, unless I am going to do it right and not have to hope nothing goes wrong.

Would anyone recommend this or have an alternative solution?
 
I'm not sure what your activities would be, but i do get the sense you worry about the realibity of the data stored. Just RAID is not enough here; filesystem corruption could occur, the RAID may be broken or multiple disks could fail at the same time due to external cause (power issues, physical trauma, fire).

The last risk can only be negated by keeping a copy of the data off-site, for example at your home where you keep a snapshot of the data each month and have to sync over the internet to get it. Since you only have to transfer the files that were modified only the first sync will be long.

The first two risks are the most important ones. You want the server to run stable without any problems arising from the disks. Ideally, you should use ZFS for this task. This filesystem is fitted with self-healing, checksumming and stores metadata on multiple disks. This should provide a formidable protection against filesystem corruption. Also, the snapshots and filesystem pools should appeal to you, as well as stuff like compression that can be set, and storing some files multiple times. Whenever one file is found bad/corrupted the other copy is used. This should be your system files.

One operating system that has the newest ZFS integrated is FreeBSD 8.0-CURRENT. This is a development release and you should not use it for the final server; but you could try and experiment with this OS within a virtualbox/vmware session or so. Discover the usefulness of ZFS and see if it suits your needs. If so, a stable release with FreeBSD with ZFS version 13 (the newest) should be released soon.

If you automatically create snapshots, you maintain a history and thus a backup of your files. So if a malicious user deletes stuff you shouldn't do, you can revert or even browse older versions of the filesystem; as if you were going back in time. These snapshots do not require additional storage space, they only store data that was changed since the last snapshot. If you want to discover storage, ZFS should prove a helping hand. Just know that not everyone deems using ZFS as safe, as it's a new and fairly complex filesystem. For me it's just working great on 8.0-CURRENT.
 
I would recommend virtuoso by sw-soft ( now parallels ) for making your poweredge / hp proliant split into VPSs with their own snapshot backups of each VPS and its contents for another from of a quick disaster recovery. Then bunch up these VPS snapshots and push them to a offsite backup in whatever form; say tapes handled by iron mountain.
 
After more research on hardware, I have decided to consider tape backup to allow me to keep tapes of my disk array in times of emergency since I live in FL. I figure, all racked tape drives are pretty expensive from Dell, so I might as well invest in the TL2000 automated library. An experiences?

As far as the MD3000 goes, I am stuck between this and the MD1000. Without knowing first hand experiences, I'd be OK with either.

I still need to research more on OS/software level decisions as suggested above. I was thinking of CentOS. I am also going to need a linux-based backup software. I have found many options and still need to read more on them. Does anyone have suggestions?
 
You might want to check out www.servondemand.com

PM me or email sales@servondemand.com if you are interested in a custom configuration with your requirements. One of the things we can offer you is off-site backup solutions for your storage system or off-site duplication so that your data is protected in the most unexpected events.
 
Data Vaulting is a pretty complex depending on what your actual goals are.

Generally if you are Tape Vaulting you need to factor in

- How long you hold vault the tape for?
- How do you manage the vaulted data? -- I.E. someone comes to you and says I need File X from Date Y, how are you going to accomodate this?
- When are you doing a media refresh or data migration to current mediums aka Moving from LTO tapes to LTO4 tapes
- How are you going to log everytime a tape has been touch and why it was touched for, for auditting purposes?

-- The only software package that I personally know that can accommodate that in a single solution is the Hiback ixT HiFreezer software suite. Expensive but accounts for all of the above factors.

In my opinion ensure data integrity I would put inplace storage replication between multiple Colo Sites.
 
What are you going to do that will make you better than all the other schmoes in the business?
 
The only unique benefit I can offer my customers is that I am local. My hardware is local and my services are local. I'm not really out to get any customer in the US/world. I want to setup a great foundation that local businesses can utilize and benefit from. With that, I will be able to show my face at my customer's offices and actually meet and greet my customers.

Also, for smaller businesses who are looking to archive single copies of files, I will be able to transfer data physically to my server on external 1TB drives/etc if need be. This will save my customer's bandwidth uploading tons of data at once for 1-2 months.

For scalable storage I have decided on the MD1000. It is cheaper and will support two additional daisy chained MD1000s.

For backups I am going to start with LTO-4 and hardware level encryption on the TL2000. Until I get more than 19.2 TB of data, this should work great. 864GB/hr should be plenty fast enough to make weekly backups. I'd probably archive/switch tapes every quarter or six months to a year.

I know I need to research more in regards to client backup software, etc. Customer number 1 needs to store 2TB of images and will SFTP them up. Easy. But I need insight on how to sell the data vault to future customers who need to backup data automatically, etc.
 
jdm71488 said:
The only unique benefit I can offer my customers is that I am local. My hardware is local and my services are local. I'm not really out to get any customer in the US/world. I want to setup a great foundation that local businesses can utilize and benefit from. With that, I will be able to show my face at my customer's offices and actually meet and greet my customers.

Also, for smaller businesses who are looking to archive single copies of files, I will be able to transfer data physically to my server on external 1TB drives/etc if need be. This will save my customer's bandwidth uploading tons of data at once for 1-2 months.

For scalable storage I have decided on the MD1000. It is cheaper and will support two additional daisy chained MD1000s.

For backups I am going to start with LTO-4 and hardware level encryption on the TL2000. Until I get more than 19.2 TB of data, this should work great. 864GB/hr should be plenty fast enough to make weekly backups. I'd probably archive/switch tapes every quarter or six months to a year.

I know I need to research more in regards to client backup software, etc. Customer number 1 needs to store 2TB of images and will SFTP them up. Easy. But I need insight on how to sell the data vault to future customers who need to backup data automatically, etc.
Click to expand...


Local will cost you dearly if you do it in house. In house backup solutions are not fondly looked upon. You need a secure, protected, and redundant facility.

Look for a colocation provider local to you.
 
You are doing all this at two widely separate physical locations in case one gets hit by fire, flood, or hurricane right?

edit: also what are you doing to ensure data security? Are you encrypting all the data per customer? Buying/creating SSL's for each?
 
I'd love to not have to have my tape backup in the same location as the servers themselves. However, I don't have a better idea that is as cost-effective. There are no other colocation sites local.

The TL2000 was going to offer encrypted backups, but the data itself was not going to be encrypted. Do you know of a way that I could encrypt the data stored on my DAS cluster per each customer?
 
As the QA person for a software company that has been putting out online backup software for over 10 years now, and other backup software for over 20 I believe. I can tell you right now SFTP, while in theory sounds like a very good solution, in the end it will come back to bite you.

A couple recommendations on 'software that doesn't totally suck' for doing what you want to do are the following:

http://us.novastor.com/products/novanet-web.html (Dislaimer, this is the company I work for)

http://www.vembu.com/

http://remote-backup.com/


As for hardware, I would strongly suggest to you that you look into Amazon's EC2 coupled with S3 storage. I know you want to see/tweak/know the hardware, just like anyone else on this site, heck we are hardware junkies. :) That being said, the benefits of Amazon's solution or others out there, is that you don't really have to worry about being geographically diverse, don't have to worry about hardware or connection issues. You just pay a fee at the end of the month for what you use. I know that RBS and Vembu, mentioned above, have AMI's available for EC2. My company currently does not have one published, but will 'soon'.


Let me just say that I came to the company I currently work for from a Backup Service provider that has made a pretty decent name for themselves. They are in a small Iowa town of 6,000 people, they started by just soliciting local businesses. This worked extremely well for them, then they started doing online signups, and some advertising via google adwords. After just a couple years they had over 3,000 users when I left. I do not know their current backup user count.

If you do contact my company about software, just mention that Nate sent you and you are from HardOCP and that he said they would take care of you. ;) As with all the companies I listed, I know the shopping cart is soo tempting to not have to deal with an actual person, but I can assure you, you can get a much better price over the phone especially with it being end of the year..
 
Help on services like S3 and EC2 is much appreciated. I looked into them initially, as well as Mosso when speaking with a Rackspace engineer, but found it to be proprietaty and somewhat expensive. Mosso, however, doesn't charge for incoming data.

Example pricing for Amazon S3 just to handle one customer at 2TB of storage.

Initial, one-time upload: 2048GB x $0.100 = $204.8
Recurring monthly: 2048GB x $0.150 = $307.00

This is not bad at all. I would be paying probably close to $1000/mn for the hardware loan and colocation. I'm not sure how much it would be if my customer decided to pull down all 2TB of data to restore a backup, etc. I'm scared of racking up excess fees. I'd have to prevent this or figure out a pricing scheme to cover actions like this.

I'd choose this route any day if I could have a good way for a client to interface with it. I didn't like Mosso because my customer couldn't SFTP data into it. And I'm trying to stay away from building a Web application and forcing customers to use that. Is this were EC2 comes in?

I'll continue to research front-end solutions.
 
EC2 is basically a giant VM host. You can spawn as many VM machines on it as you want. I believe, correct me if I am wrong, a single VM costs something like $70 a month, and it is basically your machine to do with what you please. There is a fairly sweet plugin for FireFox to control the entire distribution of machines and such. If you see that you need more CPU horsepower or whatever, you just spin up another machine. There is an entire API for this if you wanted to go about writing some auto balancing things and all that, but that is about the gist of EC2.
 
Great. So EC2 would be my 1U poweredge and S3 would take the place of my MD1000 cluster, essentially. Also, I wouldn't have to worry about backups and fooling with tapes since Amazon "should" be backing everything up, correct?

Let's say I go with the above, which I'm cool with, I think the only thing left is how to allow my customers to store data to my S3 account using my EC2 virtual server and keep everything separate per each customer.

Also, wouldn't this take quite a long time to push 2TB+ of data up there? I'd suppose it would top out my customer's upload limit on their internet connection. ;)
 
Yep, it sounds like you have it. As for the backup thing...you would be making a backup of a backup, where the user 'should' be doing a local backup too. It is sort of up to your model and how you want to market your solution on how you want to proceed that way. Put it this way, your 'storage' will be fine, but if you screw up and delete something or your users screw up and delete something, it might be worth keeping some sort of archival backup for such incidents.

Basically whatever online backup software you go with you should be able to separate the storage for your customers. That is one of the many reasons I would steer you away from just a straight SFTP solution.

Yes, it could take just a bit to get the initial upload there, but after that it should just be changes/new files. ;)
 
You must log in or register to reply here.
Back
Top