• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Script to check data integrity?

N

nocloud

n00b
Joined
Apr 28, 2011
Messages
18
Does anybody know the best way to check data integrity on Linux? In my situation, I need to keep approximately 12TB of data constantly online (4x3TB). The system will be JBOD since each 3TB disk holds static data that is backed up offline on another 3TB disk.

Is there a way in Linux to periodically scan all 12TB of data to check for data integrity? I would like to make sure that all bytes are still readable and if one file is unreadable, to give a warning message so I can delete the bad file and replace it with a copy from backup.

I'm concerned about sectors going bad and individual files within the 12TB slowly going bad and becoming inaccessible when they are requested.

Also, don't suggest things like XFS, etc. I need maximum compatibility, so Centos 6 with ext4.
 
You can make a SHA512 hash of every file and store it in a BerkeleyDB, just make sure to make two BerkeleyDBs so you can pick up rotting checksums in the DB too.
 
Code: 
find /media/foo -type f -exec sha512sum {} \; > file_sums

Code: 
sha512sum --check file_sums
 
nice, forgot this can be done using checksums. hopefully the performance is tolerable for 12TB of data....
 
nocloud said:
nice, forgot this can be done using checksums. hopefully the performance is tolerable for 12TB of data....
Click to expand...

It'll take a lot longer with 12TB of small files than 12TB of big files. If most files are 100's of mb or larger you should get close to the sustained transfer rate of your raid.
 
It would be about 4 times faster if you have enough CPU power and IO bandwidth, and start 4 parallel jobs, one for each drive. Also, sha512 is possibly overkill. If you have enough CPU power, fine, that is a good choice. But if it is taking up too much CPU, you could try a less computationally intensive checksum.
 
Too bad you're not using a filesystem that does this automatically :/
 
extide said:
Too bad you're not using a filesystem that does this automatically :/
Click to expand...

nothing too worry about. Have been using Linux ehem... RHEL/Centos/ubuntu/debian.
I do not see a single corrupted file (protected by UPS)

even I am using a filesystem that do checksum on the fly without protected by UPS.. any file can be corrupted when the power source got spikes :p.(assuming a filesystem can resume from cache drive:)).

I'm running H/W raid, mdadm RAID1&6, and zfs (IO, would stick with zfslinux when mature enough)
 
You might want to use par2 records instead of SHA, you could also build a recovery record where any failed/changed data could be repaired in addition to just being discovered.
 
I hate to sound like a ZFS fanboy, but I have to admit running a scrub once a week or so would do the job.
 
olavgg said:
You can make a SHA512 hash of every file and store it in a BerkeleyDB, just make sure to make two BerkeleyDBs so you can pick up rotting checksums in the DB too.
Click to expand...
Yes, you need to be able to detect corruption in the checksum file too. Have a backup of the checksum file somewhere.

And you need to have a backup of all your data, so you can restore a corrupted file from somewhere. Hope that your backup has no corruption.

If you update your data, you need to immediately update the checksum file. Or it will go out of sync. Which means it will be hell to compare files with the correct checksums. And if you get several corruptions, then you need to resolve which checksum file is newest, and which file is the newest. And then copy files back and forth, and update checksum file.

Every once in a while, you need to check your backup data and your current data, because of bit rot. With time, bit flips will occur spontaneously. So you need to scan your backup too.

I dont know of a script that does all of this above. Does anybody know? It would be a killer script that everybody wants! Why are there no such script out there? Anybody wants to make one? That automatically detects and restores corrupt files, and resolves all dependencies?
 
brutalizer said:
I dont know of a script that does all of this above. Does anybody know? It would be a killer script that everybody wants! Why are there no such script out there? Anybody wants to make one? That automatically detects and restores corrupt files, and resolves all dependencies?
Click to expand...

There already is one - ZFS :) ;)
 
Billy_nnn said:
There already is one - ZFS :) ;)
Click to expand...
Yes, ZFS does all that I mentioned automatically which is the single reason people are using it. But you could develop such functionality yourself using a script, or a program. Are there any such software? Then you would not have to stay with ZFS, you could choose any filesystem, even NTFS.

It shouldn't be that hard to develop such a software? Or am I mistaken?
 
brutalizer said:
Yes, ZFS does all that I mentioned automatically which is the single reason people are using it. But you could develop such functionality yourself using a script, or a program. Are there any such software? Then you would not have to stay with ZFS, you could choose any filesystem, even NTFS.

It shouldn't be that hard to develop such a software? Or am I mistaken?
Click to expand...

I suppose its impossible.
Storage changes content very often. A script that needs to asnc read/write/verify checksums on current filesystems in the
Multi-Terabyte range with optionally millions of files must fail until its embedded in the filesystem .

Next is, that it needs to check open files. How should that be done on a software application level.
I would say, not possible without a modern filesystem like btrfs (not ready), WinREFS (we will see) or ZFS
 
It's viable on filesystems which don't have a lot of updates - media collections for example, but for general purpose filesystems, then I'd agree that it's a non-starter really.
 
http://snapraid.sourceforge.net/

snapraid keeps checksums of all your data, and also keeps parity data so you can restore any files that fail a checksum. You can run a "check" operation to test the checksums of all your data. It works on most OSs, and all filesystems (except the parity drives have some limitations on filesystem choice).

It is not realtime (you need to run sync whenever you want to update the checksums and parity), but that is fine for a collection of mostly static data such as the OP seems to be talking about.
 
MS has File Checksum Integrity Verifier for Windows: http://www.microsoft.com/en-us/download/details.aspx?id=11533 - scripting this is not hard, I made a PowerShell script that does the same thing (PowerShell is pretty slow, being interpreted though) at http://poshcode.org/2876 - shouldn't be too hard to translate it to other scripting languages. Nice thing about my script, is that you can check (or update the sha-512 hash) of a subset of the files, like if you just change/add one file, you don't have to recompute all the hashes to update it's hash, etc.
 
Last edited:
Aside from ZFS, there are tools for that. No need to mess with half-assed self-written shell snippets.

Google mtree or tripwire.
 
Anybody used mtree or tripwire or snapraid? Does those solutions guarantee what is in RAM, is also what is written to disk - end-to-end checksums?

Or do they just write down everything, and then apply a checksum and store the checksum, and later compare checksums?

There is a big difference between using checksums, and end-to-end checksums. Heck, all hard disks have lot of checksums to ensure data integrity and still you get corrupted data. As CERN points it out in a study on data corruption: "checksums does not do. You must use end-to-end checksums"
 
brutalizer said:
Anybody used mtree or tripwire or snapraid? Does those solutions guarantee what is in RAM, is also what is written to disk - end-to-end checksums?
Click to expand...

Yes, of course, as much as ZFS is "end-to-end". Though ZFS is obviously NOT "end-to-end" unless the data is created in ZFS. Which is not the case here, as the OP obviously is not using ZFS.
 
JoeComp said:
Yes, of course, as much as ZFS is "end-to-end". Though ZFS is obviously NOT "end-to-end" unless the data is created in ZFS.
Click to expand...
I dont understand. Do you mean that ZFS end-to-end is nothing special, compared to other checksum solutions? It is basically the same solution?
 
brutalizer said:
I dont understand. Do you mean that ZFS end-to-end is nothing special, compared to other checksum solutions? It is basically the same solution?
Click to expand...

Yes, when the data is not created with/on the ZFS filesystem, as is the case in this thread. This is obvious...not a subtle point, I should hardly have to explain it.
 
JoeComp said:
Yes, when the data is not created with/on the ZFS filesystem, as is the case in this thread. This is obvious...not a subtle point, I should hardly have to explain it.
Click to expand...
This is not obvious to me. Can you explain? I dont understand what you mean.
 
JoeComp said:
Yes, when the data is not created with/on the ZFS filesystem, as is the case in this thread. This is obvious...not a subtle point, I should hardly have to explain it.
Click to expand...

I thought the issue being discussed was not an initial check of the data, but making sure it doesn't go bad.
 
danswartz said:
I thought the issue being discussed was not an initial check of the data, but making sure it doesn't go bad.
Click to expand...

Yes, but first you have to compute checksums on the data so that you will be able to determine if it goes bad in the future. And to compute the checksums, you have to read the data from storage into RAM. And the data could get munged before you get the data safely into RAM for the computation. I feel like I am explaining the obvious here, but this could happen with ZFS, too, if your data is coming from a non-ZFS filesystem. ZFS is only "end to end" if ZFS is in control of the data from "end to end". Is this really that difficult a concept?
 
Is there some reason you have to be a condescending a**hole? Obviously the data needs to get TO a ZFS filesystem before it can be protected, we are not idiots. My point was: I thought what was being discussed was how to prevent your good data from going bad once it is ON the server. Once the data is on the ZFS filesystem, it doesn't need to be checked going forward. Unless there is some subtle nuance we are missing, in which case, feel free to clue us in.
 
Hey, calm down guys. It is wednesday today! You have to watch this, it will make you smile:
http://www.youtube.com/watch?v=KaqC5FnvAEc

This is the original:
http://www.youtube.com/watch?v=2Z4m4lnjxkY&feature=related

JoeComp said:
Yes, but first you have to compute checksums on the data so that you will be able to determine if it goes bad in the future. And to compute the checksums, you have to read the data from storage into RAM. And the data could get munged before you get the data safely into RAM for the computation. I feel like I am explaining the obvious here, but this could happen with ZFS, too, if your data is coming from a non-ZFS filesystem. ZFS is only "end to end" if ZFS is in control of the data from "end to end". Is this really that difficult a concept?
Click to expand...
We are aware of this fact, I just didnt understand your wordings. Yes, ZFS needs to have ECC RAM, for it to do its best, that is nothing new. But with ECC RAM, then your concern is alleviated?

(It was actually not obvious first. There were researchers writing a paper on ZFS data protection and they concluded that ECC is necessary. So, it was not obvious to the researchers) :)
 
JoeComp said:
What is wrong with you? I already said it was obvious, and yet you demanded that I explain the obvious. And then you insult me.
Click to expand...

You don't understand why your posts making snarky comments about 'do i really need to explain something this obvious?' could be seen as insulting? pot, kettle, black. and you still have not responded to the meat of my post. Again, I think we all understand that even a filesystem like ZFS needs to start off with known, good data. As I said before, I thought the OP was talking about data going bad, and ZFS (with ECC) absolutely will prevent that. Sure, if the data is bad as we are loading it, it can't help there, but nothing would have. I won't speak for anyone else here, but I thought it was too obvious to need saying that an error-checking FS can't protect against data that may have been corrupted before it was loaded, but in a forum like this, one often can't be sure what exactly someone else knows or is assuming, which is why I asked my first simple question, and got attitude from you.
 
danswartz said:
You don't understand why your posts making snarky comments about 'do i really need to explain something this obvious?' could be seen as insulting? pot, kettle, black. and you still have not responded to the meat of my post.
Click to expand...

Yeah, right, like I am really going to explain the obvious to you again and be insulted by you again. I did not personally insult you at all. You started with the insults. I'm done with you.
 
brutalizer said:
Anybody used mtree or tripwire or snapraid? Does those solutions guarantee what is in RAM, is also what is written to disk - end-to-end checksums?

Or do they just write down everything, and then apply a checksum and store the checksum, and later compare checksums?

There is a big difference between using checksums, and end-to-end checksums. Heck, all hard disks have lot of checksums to ensure data integrity and still you get corrupted data. As CERN points it out in a study on data corruption: "checksums does not do. You must use end-to-end checksums"
Click to expand...

on me:
have been using tripwire.
tripwire is overkilling for just checking files :p..

tripwire will pretty useful when making a backup system and do update/compare the whole system to know differences in files or links (un*x)
for me, it is kind of snapshot, where I can see the delta from current and previous.
 
JoeComp, you could do with some lessons in better communication. Just saying ;)
 
I was going to just reply to your PM but since you called me out on here I guess I will post this here.

JoeComp said:
No, my communication is fine. I just assume a minimal level of intelligence in those reading what I write. Not much. A basic high-school level is plenty.
Click to expand...

That thread is proof that your communication could be improved. A proper and intelligent person would have not resulted to snarky comments and instead simplified, or altered, the way they were speaking in order for the subject to properly understand what you are trying to say. There was an obvious breakdown in communication there, name calling, etc, very childish. I do suppose it would be wrong to entirely place the blame on you, as danswartz did post some, shall we say, 'content of poor communication' :) However it was provoked by you being condescending first.

That's just my 3rd person point of view here, and this is getting pretty far off-topic so I will stop here.
 
extide said:
That thread is proof that your communication could be improved.
Click to expand...

Your communication leaves much to be desired. You have not explained anything relevant to this thread, but have instead gone off on an off-topic (and incorrect) critique. Now, if there were an award for trolling and diverting threads off-topic, then you would get a gold star. The reason I PM'ed you was because you were off-topic, and so was my response. But you rudely posted the PM and insisted on bringing the thread even more off-topic. What great communications skills you have! (<- that is sarcasm)

To get back on topic, if your communication is so good, why have you AGAIN failed to explain the obvious to the satisfaction of the people demanding that the obvious be explained? Can you even write something on topic? Or is trolling your only talent?
 
Last edited:
JoeComp said:
Your communication leaves much to be desired. You have not explained anything relevant to this thread, but have instead gone off on an off-topic (and incorrect) critique. Now, if there were an award for trolling and diverting threads off-topic, then you would get a gold star. The reason I PM'ed you was because you were off-topic, and so was my response. But you rudely posted the PM and insisted on bringing the thread even more off-topic. What great communications skills you have! (<- that is sarcasm)

To get back on topic, if your communication is so good, why have you AGAIN failed to explain the obvious to the satisfaction of the people demanding that the obvious be explained? Can you even write something on topic? Or is trolling your only talent?
Click to expand...

I never said anything about my communication. Everyone can always learn something, you and I included.
 
You must log in or register to reply here.
Back
Top