Storing large files

S

sphinx99

[H]ard|Gawd
Joined
Dec 23, 2006
Messages
1,195
So, a lot of my storage pool (WHS) is devoted to storing very large capture files - this is mostly 1080i or 1080p captures off a Blackmagic or Aja capture card, or occasionally a huge SD lossless capture that runs a few hours. Think 40-50GB files, and a whole bunch of them.

Now I've never been comfortable letting Windows manage files of this size; things tend to go wrong and a little piece gets corrupted that nukes the whole thing and turns recovery into a very painful process. Maybe you don't know what I'm talking about, or maybe you do.

Question is - if you had data of this sort (say 10TB worth of 50GB files) that you need to store on your home server, and the data was important enough that you don't want to lose it, especially to corruption of one kind or another, what OS would you use? What filesystem? Are they all taken to be reasonably stable or do any of them have problems managing data of this volume?
 
WHS duplication doesn't strip like a RAID 5 so you won't have the "little piece gets corrupted that nukes the whole thing" situation. Deplication will literally just make another copy of the file on another hard drive.

If I had that type of data and that much space, all of which I'd need to keep, it would depend on my budget and how paranoid I was. If you already have WHS, duplication on all the data with another 5 2TB drive. The next step would maybe be 5 additional hard drives in hotswap bays and store them offsite in my office or something. Another (very expensive) step would probably be paying for a service that would take that much data, upgrading your internet to a high end business line, tape drives?
 
Last I checked, XFS was still the king for very large files. I don't know about in terms of corruption, but for manipulation and generally working with them.
 
If your comfortable with it I would think some sort of ZFS would be the best. Personally I am not, and use Win-machines. I have 2 WHS V1 boxes, one for use, one for backup. And I run checksums to make sure the files are still safe. I would think regardless of the OS or filesystem, having them duplicated in as many places as possible would be best.
 
havermeyer said:
Last I checked, XFS was still the king for very large files. I don't know about in terms of corruption, but for manipulation and generally working with them.
Click to expand...

In practice (home and at work) I do not see it being any better then ext4 and ext4 does not have the small files penalty that XFS has. Also ext4 can be shrunk.
 
You must log in or register to reply here.
Back
Top