Small business. Management of data and long term storage

H

howknows

n00b
Joined
Jan 8, 2013
Messages
17
I own a farm and I create a lot of data that I have to keep up with.

At the office I have my SQL server and a Synology DS2413+ (12 3TB drives) NAS. At home I have a Synology DS2413+ (12 2TB drives) NAS.

At my office I keep the last 2 years and the current year of data on the NAS because I find that I need it all the time. At my house I keep all the backups for 12 months and the stuff I use a lot on the NAS. Quarterly (tax time) I back up the quart to hard drives and into fire prof safe. Most of the stuff I have to keep 7 years and part of it is 20 years.

I’m going back through last years stuff checking my quarterly taxes and over all I'm just under 6TB of files for last year alone. 1.5 TB is SQL, 1.3 TB of jpeg, and 3.2 TB of docs and PDF. I scan everything in black and white and when color is needed I scan in 16 bit. Once an outgoing invoice is paid I add the payment info to the data base and delete the invoice to save space.


I know larger companies are using tape drives and paying to store everything in the cloud.
Last year when I looked at tape drives, but was talked out of it. Once I looked at the cost of tapes and a backup drive the hdd's looked better.
I was looking at the cost of long term cloud storage today and when I read the fine print I find all the extra fees that I would end up with.
What are larger companies doing and how much is it costing them a year?
 
This is a really good queston, and one I have not seen answered adequately for the SOHO segment.

I imagine multiple backups and using multiple media would be the key. I would like to see someone answer this post with a time tested solution.
 
At work (where we have 40TB+ of data that is grows daily and must not be lost) we use LTO2 tapes (over 140 now). Yes still LTO2. It cost us about $5000 for a 2 drive 24 slot autoloader in 2006. We still use that. I have looked into replacing that with an LTO4 unit since that will have read compatibility to LTO2 but we have not had the budget for that yet and the $5000+ replacement cost would pay for quite a few $25 LTO2 tapes + 20 tape turtles.
 
Last edited:
i've still read that tape + a safe is the fallback to use if you just need to preserve the data.
 
SemiLucid
I think I'm close to doing it right. The NAS and SQL server get backed up to a NAS at my house (different building). Quarterly back up and put in a safe. All data is kept at 2 buildings until it's put in a safe, I know I can do better.

drescherjm
140 tapes sound like fun. Have they ever had to do a backup or done it for fun?


Some time this year I'm going to pull all my old ATA HDD's out of the safe and move them onto new HDD's or LOT5 tapes.
Last month I had to pull some info from a ATA drive and found I did not have a computer that had ATA.
 
140 tapes sound like fun. Have they ever had to do a backup or done it for fun?
Click to expand...

I never ever do a full backup. The data is distributed over several servers and I do backups per project. The largest project is only a few TB so its all manageable. Since I carefully monitor the health of all the drives in the raid arrays and do weekly scrubs ... we have never needed to do a full restore of any project. Almost every restore has been to restore an accidentally deleted or overwritten file.

Also the backup software handles the tapes pretty well and with a 24 slot autochanger I probably have to swap tapes 1 time per month so its not like its that much hassle.
 
howknows said:
I own a farm and I create a lot of data that I have to keep up with.

At the office I have my SQL server and a Synology DS2413+ (12 3TB drives) NAS. At home I have a Synology DS2413+ (12 2TB drives) NAS.

At my office I keep the last 2 years and the current year of data on the NAS because I find that I need it all the time. At my house I keep all the backups for 12 months and the stuff I use a lot on the NAS. Quarterly (tax time) I back up the quart to hard drives and into fire prof safe. Most of the stuff I have to keep 7 years and part of it is 20 years.

I’m going back through last years stuff checking my quarterly taxes and over all I'm just under 6TB of files for last year alone. 1.5 TB is SQL, 1.3 TB of jpeg, and 3.2 TB of docs and PDF. I scan everything in black and white and when color is needed I scan in 16 bit. Once an outgoing invoice is paid I add the payment info to the data base and delete the invoice to save space.


I know larger companies are using tape drives and paying to store everything in the cloud.
Last year when I looked at tape drives, but was talked out of it. Once I looked at the cost of tapes and a backup drive the hdd's looked better.
I was looking at the cost of long term cloud storage today and when I read the fine print I find all the extra fees that I would end up with.
What are larger companies doing and how much is it costing them a year?
Click to expand...

I don't know what is more interesting:

1) You have this huge amount of data and are only now worried about it.

2) You are worried about the cost of long term storage.

3) You still have data on ATA drives and no ATA computer.

A complete backup - hard drives stored in a safe deposit box off site, could cost less than $3000. If you want to spend more you can.

---

6TB of data a year seems to indicate a very large business. One a lot larger than you present.
 
GeorgeHR
I'm not worried about my data. I think my system is fine for what I need. I'm sure it could be much better.
I'm not all that worried about the cost of long term storage, but it's nice to know the options.
I now have a ATA to USB external case and will transfer all my data from them soon.

It is nice to know what other complainers use / cost. I know I might be somewhat behind on the times on when it comes to the computer side, but farm wise I keep up much better technology wise.

I look at us to be a small business with just having 9 employees.
 
howknows said:
GeorgeHR
I'm not worried about my data. I think my system is fine for what I need. I'm sure it could be much better.
I'm not all that worried about the cost of long term storage, but it's nice to know the options.
I now have a ATA to USB external case and will transfer all my data from them soon.

It is nice to know what other complainers use / cost. I know I might be somewhat behind on the times on when it comes to the computer side, but farm wise I keep up much better technology wise.

I look at us to be a small business with just having 9 employees.
Click to expand...

Your comment "I was looking at the cost of long term cloud storage today and when I read the fine print I find all the extra fees that I would end up with." seems to be at odds with your first paragraph.

I told you what companies pay for long term storage. That takes care of paragraph 2.

Paragraph 3 takes a bit of math from me. We had about 80 lineal feet of files in crates and another 50 lineal feet of files on shelves. I scanned them in and had less than 20GB of PDF files. You have 6TB of PDF files - about 300 times what I scanned. That is a lot of PDFs. More than a business with 9 employees should generate.

So I told you how much it costs to store your data and as a bonus I told you that you generate too much data.
 
You could try out Backblaze or some similar provider, though it would probably take quite a while to upload all of that to the cloud....
http://www.backblaze.com/business.html
I believe these are one of the field cloud backup companies that bill out at a flat rate per computer, regardless of upload / download quantities.
 
That is an insane amount of data for a farm IMO. I'm not saying you aren't generating this amount of data but you might want to make sure you aren't doing full/complete backups on a daily basis rather than a differential or incremental. One of the companies I was working for was doing this and had way to much storage space and backup time was excessive and network use was extreme.
 
Question on the side: How do you get over 3TB of documents? Do you scan in 2400 DPI?

A greyscale A4 page is ~8.5MB uncompressed. That's with 300 DPI which is still plenty for OCR. I could store nearly 400k pages in 3.2TB. And remember it's uncompressed.

So do you have that many pages or are you maybe not aware of your DPI settings?
 
GeorgeHR
Sorry I did not mean to step on your toes...

Scout
Most of the flat rate per computer ones have a limit, but I will look at that one.

Yes tankman.
I spend a lot of time dealing with paperwork and my wife dose paperwork most of her day at work also. I don't think it's an issue of the types of backups, but I will check it today.

TCM
I do not think I ever changed the DPI settings. Something to look at today. After searching the net for more info I also find I need to switch from PDF search able to PDF normal.
I don't think I ever thought about theses settings.

I think I'm getting around 200k (estimation) pages in 3.2 TB. I don't know if I compress the hard drives when I do long term storage.
 
Going through my scanner setting my DPI was set at 1200. I changed it to 300 black and white. The quality is fine and takes up a fraction of the space. Thanks TCM that helped out a lot. I got a few more days of getting my end of the year taxes ready and than I have a 15x12x10” file box of paper work to scan form the 750 cattle I took owner ship of last week.

I did find a few duplicates of a few files, but they had different modified dates on them.
 
howknows said:
Going through my scanner setting my DPI was set at 1200. I changed it to 300 black and white. The quality is fine and takes up a fraction of the space. Thanks TCM that helped out a lot. I got a few more days of getting my end of the year taxes ready and than I have a 15x12x10” file box of paper work to scan form the 750 cattle I took owner ship of last week.

I did find a few duplicates of a few files, but they had different modified dates on them.
Click to expand...

You did not step on my toes.

TCM wrote "A greyscale A4 page is ~8.5MB uncompressed. That's with 300 DPI which is still plenty for OCR."

Just looking at one of our files scanned at 300x300 B&W, 8MB is 23 pages.

Going from 1200x1200 to 300x300 reduces your storage requirements by a factor of 16.
 
So I have a couple questions for you that I want you to think about.

How much is your data worth to you that you want to backup?
How much would it cost to recreate the data?
Do you depend on the historical data in your day to day working?

Now you say you keep up with your farm equipment, but are trying to skimp on the data that runs your farm. Does that add up?

A Dell PowerVault 124T LTO5 will come in around $5k or so with 16 tape slots, and will last you for years. Somewhere around $50/tape, and then store them in a safety deposit box at the bank once a month or so. That seems like pretty cheap insurance to me.
 
75% of the data I back up is for NAIS, COOL, FDA and the USDA regulations. It's not worth much to me.
If I lost the data I would only recreate the data I need for IRS. The rest of it I would worry about when the USDA or FDA came to my door.
I use some of the data for animal growth rates, but could live with out it.

I do not think I have skimp that much on computer technology. I have 2 NAS's in 2 different buildings with fiber between the buildings. Data is automatically backed up to both of them. Every few months the new data is copied to hard drives and put into a fire proof safe.
I do feel that I could improve on what I’m doing.

I have not gone to tape drives yet because of the amount of press they have gotten over time. Once I have read into the bad press I can clearly see that a lot of the bad press comes for miss use and bad management of the tapes.



I'm sorry I came and asked a simply question. If you also want to do a fecal test on me than let me know.
 
I meant no disrespect at all!

I work in the backup industry and these are the types of questions that we basically have to ask in order to determine what type of backup solution to provide.

I don't know what kind of bad press you have seen about tape, but tape is very reliable. Cloud backup is also good, if you know what you are getting into and like you said read the fine print. :)
 
Can LTO5 be obtained cheaper now that LTO6 has hit the market? Are refurb/used LTO5 drives out of the question?
 
I have been looking at a 8 or 12 slot LOT-5. From what I have read (I could be wrong) it would not be the best solution for me with the current hardware I already have. I would also have to buy a server to feed it.

If I went with a tape drive it would be much cheaper to build a desk top around an internal tape drive.

When it comes down to it I plan on doing something in the next few months, but I’m still not sure what it will be yet. I see a few different options out there and I don't know witch way I want to go yet.
 
You could probably reduce your data consumption 1000% if you converted all your scanned documents into compressed PDF. Adobe Acrobat is very efficient at this. Make sure you are using B&W and not greyscale.

I would avoid filesystem compression such as the option in Windows Explorer.
 
You must log in or register to reply here.
Back
Top