home NAS suggestions?

[ochadd]

Somehow I went from wanting a cheap ~100 nas box to store misc docs and music to wanting something like a HP N40L. This is why I try to stay away from [H] but at the same time why I love it here.

After reading all the replies I wouldn't mind having the N40L and storing my photos there as well. Now you have me second guessing my raid DAS that I used for my main photo backup.

This happens to me all the time here... I was on Newegg looking at a $200 Synology NAS after this thread started and now I'm debating building a FreeNAS and/or Server 2008r2 box.

 

[blastingcap]

Somehow I went from wanting a cheap ~100 nas box to store misc docs and music to wanting something like a HP N40L. This is why I try to stay away from [H] but at the same time why I love it here.

After reading all the replies I wouldn't mind having the N40L and storing my photos there as well. Now you have me second guessing my raid DAS that I used for my main photo backup.

Yeah when I first joined [H] I think I had a 4670 and 22" 1650x1080 and now I have multiple 7970s and 5760x1080 w/ an X-Rite color calibrator.

But [H] isn't even the worst. On DPReview I went from a 35mm compact film camera to my massive DSLR + RX100 + tripod and flash setup today. On the plus side I felt like I should use that gear and not let it go to waste, so I became a much better photographer.

 

[merlin87]

lets not talk about cameras please because this purchase is using my money for my canon 16-35II rofl

 

[brutalizer]

I don't see how something like FlexRAID isn't really safe.

Every file is checksummed. If a file gets corrupt due to non-ECC memory, or due to a bad sector, wouldn't the checksum always detect a difference? When that happens FlexRAID emails me the file(s) names that don't match and then I restore them from parity or restore them from the cloud in the worst case.

Is this solution really no good and just the ticking time bomb that you seem to say it is?

This solution is clearly better than no checksumming. But as CERN writes in a big study on data corruption "checksumming is not enough". "Data corruption is found on even very expensive Enterprise storage systems". I can try to google this study for you, if you wish.

With that said, it has not been investigated how good Flexraid data protection is. As CERN says, adding some checksums is not enough. Hard disks have lot of error correcting checksums all over the surface, but still there are corruption. There are phantom writes and misdirected reads and all sorts of weird errors.

We need comp sci research on how safe Flexraid is, but researchers have said that NTFS, ext3 etc are not safe. And Flexraid uses NTFS, ext3, etc. (Researchers say that ZFS is safe). There is no research on Flexraid, that is the reason I will not say it is safe, because the underlying filesystem is not safe, and there are lot of weird errors that checksumming does not catch. For isntance, how do handle metadata corruption? Data is checksummed, but is the metadata? Windows ReFS checksums only the data, but the metadata is not checksummed. Just adding checksums is not enough. You need special end-to-end checksums and also design the system to catch all sorts of errors (data, metadata, etc), just as ZFS does.

But I would trust more in a solution that checksums everything, yes. But making a MD5 sum of a file takes long time. As soon as you see a movie in Flexraid, it will take some seconds before checksumming is done? Or is the checksum done in like, 0.01 second? If checksumming is that fast, then it is not safe, they cheat and cut corners. There is a reason ZFS uses lot of cpu and is slow (unless you have lot of RAM and CPU).

For me, I trust ZFS most. Then a solution that checksums like Flexraid. Then a solution that does no checksumming, i.e. normal systems. I believe for a media system, Flexraid is safe enough because it does some rudimentary checksums, but does not cover the whole spectrum of errors that can occur.

 

[SirMaster]

This solution is clearly better than no checksumming. But as CERN writes in a big study on data corruption "checksumming is not enough". "Data corruption is found on even very expensive Enterprise storage systems". I can try to google this study for you, if you wish.

With that said, it has not been investigated how good Flexraid data protection is. As CERN says, adding some checksums is not enough. Hard disks have lot of error correcting checksums all over the surface, but still there are corruption. There are phantom writes and misdirected reads and all sorts of weird errors.

We need comp sci research on how safe Flexraid is, but researchers have said that NTFS, ext3 etc are not safe. And Flexraid uses NTFS, ext3, etc. (Researchers say that ZFS is safe). There is no research on Flexraid, that is the reason I will not say it is safe, because the underlying filesystem is not safe, and there are lot of weird errors that checksumming does not catch. For isntance, how do handle metadata corruption? Data is checksummed, but is the metadata? Windows ReFS checksums only the data, but the metadata is not checksummed. Just adding checksums is not enough. You need special end-to-end checksums and also design the system to catch all sorts of errors (data, metadata, etc), just as ZFS does.

But I would trust more in a solution that checksums everything, yes. But making a MD5 sum of a file takes long time. As soon as you see a movie in Flexraid, it will take some seconds before checksumming is done? Or is the checksum done in like, 0.01 second? If checksumming is that fast, then it is not safe, they cheat and cut corners. There is a reason ZFS uses lot of cpu and is slow (unless you have lot of RAM and CPU).

For me, I trust ZFS most. Then a solution that checksums like Flexraid. Then a solution that does no checksumming, i.e. normal systems. I believe for a media system, Flexraid is safe enough because it does some rudimentary checksums, but does not cover the whole spectrum of errors that can occur.

Yeah this all makes sense. I guess for me it still seems fine.

I merely need checksumming to just tell me when there is something wrong. The checksumming process is not realtime. I run it once a month and it takes something like 15 hours to go through my 10TB of data.

All I need from this process is the output. I simply want it to tell me if any of the checksums have changed since last time it ran. If any have changed, then I will check the file myself to see if it's corrupt or if it's the checksum that became corrupted. If the file is corrupt, then I will restore it from parity. If the checksum is corrupt, then I will make a new checksum.

What's the chances of both the file and checksum for that file becoming corrupt? I do not think that it's a very high chance. But still this is why my whole array is also stored in the cloud with CrashPlan. If i detect a checksum problem and cannot recover it myself via FlexRAID, then I will simply download a copy from my online backup and replace the file and redo the checksums.

This seems safe to me for a media server. Sounds like more work probably, but I have yet to have this happen and can't imagine it could happen that often really.

I can't imagine a scenario where all 3 things, my data, and my checksum/parity, and my crashplan backup all become corrupt for the same file at the same time. Thus I can't really imagine a scenario where I lose any files.

Without the cloud backup FlexRAID is just a normal array system and is definitely subject to these possibly corruptions however rare they probably are. But with a cheap backup ($48/year) for crashplan I think it makes my overall solution incredibly resilient and it works for me.

I would consider ZFS, but I really wanted a Windows server. Also, I have no way of moving to ZFS since you need to format the disks to add them to the ZFS array. I have no place to temporarily store my 10TB of data. With FlexRAID, you can add disks with existing data into the array and it was one of the prime reasons for me to choose it.

 

[GeorgeHR]

The issue isn't that no one has ever stored data reliably. It's that files can get corrupted and unless you've got a cronjob running or some script checklng checksums you have no idea if you have what you think you have.

I think I have a great product idea for you.

People spend a lot of effort making hard drive data unrecoverable. 2000bit encription is not sufficient. Writing over every sector multiple times is not sufficient.

But your concept of using bit rot to change a couple bits on a hard dirve and making all the data unrecoverable seems to lead to an efficient method of protecting data.

I think we can raise enough funds to produce a marketable product in short order.

I am in for $.02

 

[brutalizer]

Yeah this all makes sense. I guess for me it still seems fine.

I merely need checksumming to just tell me when there is something wrong. The checksumming process is not realtime. I run it once a month and it takes something like 15 hours to go through my 10TB of data.

All I need from this process is the output. I simply want it to tell me if any of the checksums have changed since last time it ran. If any have changed, then I will check the file myself to see if it's corrupt or if it's the checksum that became corrupted. If the file is corrupt, then I will restore it from parity. If the checksum is corrupt, then I will make a new checksum.

What's the chances of both the file and checksum for that file becoming corrupt? I do not think that it's a very high chance. But still this is why my whole array is also stored in the cloud with CrashPlan. If i detect a checksum problem and cannot recover it myself via FlexRAID, then I will simply download a copy from my online backup and replace the file and redo the checksums.

This seems safe to me for a media server. Sounds like more work probably, but I have yet to have this happen and can't imagine it could happen that often really.

I can't imagine a scenario where all 3 things, my data, and my checksum/parity, and my crashplan backup all become corrupt for the same file at the same time. Thus I can't really imagine a scenario where I lose any files.

Without the cloud backup FlexRAID is just a normal array system and is definitely subject to these possibly corruptions however rare they probably are. But with a cheap backup ($48/year) for crashplan I think it makes my overall solution incredibly resilient and it works for me.

I would consider ZFS, but I really wanted a Windows server. Also, I have no way of moving to ZFS since you need to format the disks to add them to the ZFS array. I have no place to temporarily store my 10TB of data. With FlexRAID, you can add disks with existing data into the array and it was one of the prime reasons for me to choose it.

Im with you. The chances of corruption in numerous places at the same time is so small you dont need to care about it. But of course, if you are storing important data, then you should take no chances. But for a home media server, I think Flexraid is perfectly fine, as this Comp... guy declares with great energy. I do hope that FlexRaid is a pooled solution, though. It would be a pain to handle all disks individually. Specially if you have 10TB or more data.

I think I have a great product idea for you.

People spend a lot of effort making hard drive data unrecoverable. 2000bit encription is not sufficient. Writing over every sector multiple times is not sufficient.

But your concept of using bit rot to change a couple bits on a hard dirve and making all the data unrecoverable seems to lead to an efficient method of protecting data.

I think we can raise enough funds to produce a marketable product in short order.

I am in for $.02

Actually, this is not so weird as it sounds. ZFS devs where considering a solution where ZFS tried all bit combinations to recover from data corrutpion, but it used to much time and power scrapped the idea. And besides, ZFS already has good protection.