• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Script to check data integrity?

JoeComp said:
No. And if you ask the same question a third time, the answer will still be no.
Click to expand...
Earlier, I asked about if ECC would solve your concerns, and you said no. Now I asked about ECC + ZFS, and you said no.

How about you explain a bit more, I dont really understand what is so obvious to you? If you explain, then we dont have to ask a lot about this thing, that is so obvious to you, but to no one else?
 
brutalizer said:
How about you explain a bit more, I dont really understand what is so obvious to you?
Click to expand...

How about if you read and think a bit more so you can see what is not only obvious, but has already been explained in this thread?
 
JoeComp said:
What is wrong with you? I already said it was obvious, and yet you demanded that I explain the obvious. And then you insult me.
Click to expand...

Yeah but you deserve it - you have a distinct lack of manners in just about every comment.
 
brutalizer said:
Earlier, I asked about if ECC would solve your concerns, and you said no. Now I asked about ECC + ZFS, and you said no.

How about you explain a bit more, I dont really understand what is so obvious to you? If you explain, then we dont have to ask a lot about this thing, that is so obvious to you, but to no one else?
Click to expand...

What he's saying is that if you have a server running ZFS, with ECC memory, you can still put corrupt data on it in the first place. Essentially, ZFS with ECC memory on hardware that isnt broken can ensure that you get back exactly what you wrote. The issue remains with writing bad data in the first place. Now if you say, "I know the data started out as good" then with ZFS it will stay the same so long as it stays solely on the ZFS box. For example, if you have a windows box putting data onto the ZFS box, and manipulating the data on that box then those actions could introduce errors that ZFS would not consider errors, and thus circumvent the ZFS checksums and ECC memory protection.

As far as the OP's question goes, what you would want is some sort of hash of the file at the beginning, an 'initial hash' and then some mechanism to periodically check the actual hash against the initial hash. You'd probably need to roll your own solution to get something like that. I don't know of anything off the top of my head that does that.
 
extide said:
What he's saying is that if you have a server running ZFS, with ECC memory, you can still put corrupt data on it in the first place. Essentially, ZFS with ECC memory on hardware that isnt broken can ensure that you get back exactly what you wrote. The issue remains with writing bad data in the first place. Now if you say, "I know the data started out as good" then with ZFS it will stay the same so long as it stays solely on the ZFS box. For example, if you have a windows box putting data onto the ZFS box, and manipulating the data on that box then those actions could introduce errors that ZFS would not consider errors, and thus circumvent the ZFS checksums and ECC memory protection.
Click to expand...
Really? Is this what JoeComp means? No, I think you have misunderstood him because that would be really weird. The reason I dont understand JoeComp, and kept asking again and again, is because that such a request would be just too strange, let me see if I understand your interpretation of JoeComp's posts correctly:
"JoeComp claims that if the data that ZFS gets, is already corrupted, then ZFS can not fix such errors?"
Such a request would be just stupid, and I doubt your interpretation. I mean, how in earth could any filesystem repair already corrupt data?

For a filesystem to repair, say C++ files, it has to have knowledge of C++, it must have an C++ compiler built in, to be able to debug C++ source files and correct it automatically. But say that the C++ file has a corruption where a line is lost. How can the filesystem know if the line should be "color the window black", or "color the window white" or which color? For this to be possible, the built in C++ compiler, must be able to read minds, it has to have psychic powers, otherwise the filesystem can never figure out which color the window was meant to have. Say that several C++ files are heavily corrupted, then the filesystem must use its psychich powers to be read the minds of the developers, and restore the corruption.

And if the filesystem needs to be able to repair corrupt Photoshop data file, the filesystem must have Photoshop built in, with psychich powers. And the filesystem should have all different versions of Photoshop built in, to restore correct file version.

And the filesystem must have MS office built in, all versions, so he understands MS Office data files and repair those files. And have Diablo3 built in, so he can repair D3 files. etc

The filesystem must have all software ever constructed built in (or a database describing each file) and in addition to that, it must be able to read minds of living developers, and dead developers.

Now, is this is a feasible request of a filesystem? Is this a clever request? Hardly. Such a requester must be IT illiterate or just dumb. This is "obviously" impossible to achieve. Psychic powers? All software ever constructed, built in?

My first interpretation of JoeComp was this too, but no one is that dumb, so I figured out I must have misinterpreted him. And because he will not answer to questions, we will never find out exactly what he meant.




As far as the OP's question goes, what you would want is some sort of hash of the file at the beginning, an 'initial hash' and then some mechanism to periodically check the actual hash against the initial hash. You'd probably need to roll your own solution to get something like that. I don't know of anything off the top of my head that does that.
Click to expand...
ZFS does that. But other than that, I dont know how good tripwire or anything else works. But snapraid(?) is claimed to have a solution similar to ZFS?
 
^^
as I posted for tripwire. this tools is remarkable helpful for checking the delta( not the file only. the rights aka drwxl, system changes), where very useful troubleshooting "hijacked" system or unstable system after the major update
tripwire can revert back to previous state (files, system changes,libraries, and many others)
does this zfs support?just throwing an Idea...
as I know, non built-in "tripwire" tools in any filesystem.

tripwire is unix flavor friendly.

tripwire create a kind of repo to save the data and changes. a dedicated shared/NFS/SSH can be used. I personally use NFS :D.
 
mwroobel said:
<=- Looks at my phone... I thought it was Monday today :)
Click to expand...

I went to Moes yesterday to get Joe Moes Monday buritos special for $6 hehehehehe.
and some people are really hate Moes... hehehhe

sorry not stick with the topic, just try to cool down the heat. :p
 
cantalup said:
^^
as I posted for tripwire. this tools is remarkable helpful for checking the delta( not the file only. the rights aka drwxl, system changes), where very useful troubleshooting "hijacked" system or unstable system after the major update
tripwire can revert back to previous state (files, system changes,libraries, and many others)
does this zfs support?just throwing an Idea...
as I know, non built-in "tripwire" tools in any filesystem.
Click to expand...
Yes, ZFS supports this. It is called Snapshot. You can roll back to earlier states. If your zpool becomes corrupt, you can rollback to an earlier state. If that state is also corrupt, you can rollback to an even earlier state, etc. You use "zpool import -F"
 
brutalizer said:
Really? Is this what JoeComp means? No, I think you have misunderstood him because that would be really weird. The reason I dont understand JoeComp, and kept asking again and again, is because that such a request would be just too strange, let me see if I understand your interpretation of JoeComp's posts correctly:
"JoeComp claims that if the data that ZFS gets, is already corrupted, then ZFS can not fix such errors?"
Such a request would be just stupid, and I doubt your interpretation. I mean, how in earth could any filesystem repair already corrupt data?

For a filesystem to repair, say C++ files, it has to have knowledge of C++, it must have an C++ compiler built in, to be able to debug C++ source files and correct it automatically. But say that the C++ file has a corruption where a line is lost. How can the filesystem know if the line should be "color the window black", or "color the window white" or which color? For this to be possible, the built in C++ compiler, must be able to read minds, it has to have psychic powers, otherwise the filesystem can never figure out which color the window was meant to have. Say that several C++ files are heavily corrupted, then the filesystem must use its psychich powers to be read the minds of the developers, and restore the corruption.

And if the filesystem needs to be able to repair corrupt Photoshop data file, the filesystem must have Photoshop built in, with psychich powers. And the filesystem should have all different versions of Photoshop built in, to restore correct file version.

And the filesystem must have MS office built in, all versions, so he understands MS Office data files and repair those files. And have Diablo3 built in, so he can repair D3 files. etc

The filesystem must have all software ever constructed built in (or a database describing each file) and in addition to that, it must be able to read minds of living developers, and dead developers.

Now, is this is a feasible request of a filesystem? Is this a clever request? Hardly. Such a requester must be IT illiterate or just dumb. This is "obviously" impossible to achieve. Psychic powers? All software ever constructed, built in?

My first interpretation of JoeComp was this too, but no one is that dumb, so I figured out I must have misinterpreted him. And because he will not answer to questions, we will never find out exactly what he meant.





ZFS does that. But other than that, I dont know how good tripwire or anything else works. But snapraid(?) is claimed to have a solution similar to ZFS?
Click to expand...

You mis-understood me here, I never said ZFS would fix corrupt data. I said that ZFS would give you back exactly what it was given. If it was given errors then you will get errors back. ZFS CAN repair files that get corrupt on disk AFTER they have been written (correctly) by ZFS as ZFS includes redundancy. It's not magic. JoeComp's point was that adding ECC memory is only PART of the protection scheme, just like ZFS itself, that's only PART of the protection scheme.
 
brutalizer said:
Yes, ZFS supports this. It is called Snapshot. You can roll back to earlier states. If your zpool becomes corrupt, you can rollback to an earlier state. If that state is also corrupt, you can rollback to an even earlier state, etc. You use "zpool import -F"
Click to expand...

Nope. Snapshot is a different beast.
Check Tripwire site. You can find. Many interesting scenarios.
Tripwire is more more powerful than snapshot. How about finding out some incompatible s libraries after amjor update. Can snapshot do that?
Can snapshot restores. Certain object files and check dependencies?
 
extide said:
You mis-understood me here, I never said ZFS would fix corrupt data. I said that ZFS would give you back exactly what it was given.
Click to expand...

I think you explained the obvious as well as I did. But it did not seem to get the point across, did it? You can see why I stopped trying.
 
extide said:
You mis-understood me here, I never said ZFS would fix corrupt data. I said that ZFS would give you back exactly what it was given. If it was given errors then you will get errors back. ZFS CAN repair files that get corrupt on disk AFTER they have been written (correctly) by ZFS as ZFS includes redundancy. It's not magic. JoeComp's point was that adding ECC memory is only PART of the protection scheme, just like ZFS itself, that's only PART of the protection scheme.
Click to expand...
So what is his point? That ECC RAM is a necessary step, to provide data integrity? Yes, everybody knows that nowadays (after the research paper, but that was not obvious some years ago). So basically, JoeComp just reiterated what everybody says? For data integrity, use a filesystem with end-to-end checksums and ECC RAM? Great info, that was news to us here, thanx for enlightening us. That was the obvious part? Why not just write that, instead of confusing people here? The next time we have to communicate with JoeComp, we will ask you, who understands his incomprehensible posts, while noone else does.



cantalup said:
Nope. Snapshot is a different beast.
Check Tripwire site. You can find. Many interesting scenarios.
Tripwire is more more powerful than snapshot. How about finding out some incompatible s libraries after amjor update. Can snapshot do that?
Can snapshot restores. Certain object files and check dependencies?
Click to expand...
I checked Tripwire site, there was many links. Can you post those interesting scenarios?
 
brutalizer said:
So what is his point? That ECC RAM is a necessary step, to provide data integrity? Yes, everybody knows that nowadays (after the research paper, but that was not obvious some years ago). So basically, JoeComp just reiterated what everybody says? For data integrity, use a filesystem with end-to-end checksums and ECC RAM? Great info, that was news to us here, thanx for enlightening us. That was the obvious part? Why not just write that, instead of confusing people here? The next time we have to communicate with JoeComp, we will ask you, who understands his incomprehensible posts, while noone else does.
Click to expand...

No, the point was that using ZFS, EVEN WITH ECC, can not always guarantee data safety.
 
brutalizer said:
I checked Tripwire site, there was many links. Can you post those interesting scenarios?
Click to expand...
tutorial is spreading on the net

start with http://www.tripwire.org/, a
you will be given 3 selections. pick on tripwire open source selection that links to http://sourceforge.net/projects/tripwire/

the last 2 selections are commercial products with GUI, enhancement and supports

open source GPL tripwire is a CLI only as I know, and users/forums/mailinglists support only :p
 
extide said:
No, the point was that using ZFS, EVEN WITH ECC, can not always guarantee data safety.
Click to expand...
Is that the point? Stupid. No solution on earth can always guarantee data safety, no matter if you pour in billions of USD worth of research. You can almost never guarantee anything:
-Can you guarantee that you are not a simulation in a big computer? No.
-Can you guarantee that the earth not will blow in the next hour? No.
-Can you guarantee death? But if you are a simulation in a big computer, there might be no death?

So all those incomprehensible posts was about this deep fact "you can never guarantee anything". Great info, thanks for sharing. And then calling people stupid for not being able to express himself clearly, that is not just really clever. But thanks for interpreting.


If we talk about realistic and achievable goals with data integrity, CERN have made surveys and concluded that even very expensive Enterprise storage systems using checksums might not offer good data integrity. CERN says:
-silent data corruption is a fact of life
-checksums is not enough, you need to use end-to-end checksums, so ZFS has a point.
And CERN is now switching to ZFS for long term storage of all particle data, to get best data protection.
 
JoeComp said:
I think you explained the obvious as well as I did. But it did not seem to get the point across, did it? You can see why I stopped trying.
Click to expand...
"You can lead a man to knowledge, but you can't make him think."

----
[ "You can lead a horticulture, but you still have to pay her." ]
 
extide said:
What he's saying is that if you have a server running ZFS, with ECC memory, you can still put corrupt data on it in the first place. Essentially, ZFS with ECC memory on hardware that isnt broken can ensure that you get back exactly what you wrote. The issue remains with writing bad data in the first place. Now if you say, "I know the data started out as good" then with ZFS it will stay the same so long as it stays solely on the ZFS box. For example, if you have a windows box putting data onto the ZFS box, and manipulating the data on that box then those actions could introduce errors that ZFS would not consider errors, and thus circumvent the ZFS checksums and ECC memory protection.

As far as the OP's question goes, what you would want is some sort of hash of the file at the beginning, an 'initial hash' and then some mechanism to periodically check the actual hash against the initial hash. You'd probably need to roll your own solution to get something like that. I don't know of anything off the top of my head that does that.
Click to expand...

Yes, I understand this. Obviously, you need to get good data on the box, but as I said already, I had thought the OP's question pertained to making sure the data hasn't gone bad since then. If that wasn't what was being asked (or not all of it), that is why I asked for a clarification as to what the problem statement was. And I got snarkiness about 'do i really need to explain something so obvious?' Issues like allowing a remote client (likely without ECC) to manipulate the files on the storage server (over TCP/IP which CAN have undetected checksum errors) obviously complicates all of this.
 
brutalizer said:
Is that the point? Stupid.
Click to expand...

If you feel that way, one wonders why you brought up the point in the first place (are you calling yourself stupid?). The subject of "end to end checksums" that you brought up (post #20) is hardly relevant to the OP's question about protecting TBs of already existing static data that apparently are not currently protected by checksums.
 
JoeComp said:
If you feel that way,
Click to expand...
I dont feel that way. It IS stupid. No one can guarantee anything. Period. If you believe something else, I suggest you go to university and learn something.

Can you guarantee you are not a computer simulation? No you can not.

Do you know of ANY storage solution that guarantees data integrity? There are no one. If somebody claim it, you are stupid if you believe such bull.

(Isn't it nice with all this snarkyness, it makes you feel warm and cozy inside, doesn't it? You really like it, when people treat you like I do, don't you)
 
brutalizer said:
I dont see those interesting scenarios? Do you have a link to them?
Click to expand...

I give you a starting point and can not give you everything that you want.
I just thrown some examples on my previous posting
you have to read!... :p
the information is on the net.

if you are working on maintaining servers (at work/home), any wicked situation can happens.
 
JoeComp said:
If you say it is stupid for you to bring up "end to end checksums" in this thread, then why did you do it?
Click to expand...
I am saying it is stupid to believe that any filesystem can guarantee data integrity. That is unrealistic. Do you agree on this? Or do you believe any storage solution on earth, can guarantee data integrity?
 
cantalup said:
I give you a starting point and can not give you everything that you want.
I just thrown some examples on my previous posting
you have to read!... :p
the information is on the net.
Click to expand...
Yes, I went to your link
http://sourceforge.net/projects/tripwire/

And you said there are lot of scenarios described:
"Tripwire is more more powerful than snapshot. How about finding out some incompatible s libraries after amjor update. Can snapshot do that?
Can snapshot restores. Certain object files and check dependencies? "

I want to read more about incompatible libraries, dependency checking, etc. I am always looking for a better storage solution. If it can do this, then I might want to use tripwire. Where are all those scenarios you talked about? Are there any reports? Articles? Research papers? Or do you want me to google for something vaguely described by you? If there are any links with scenarios, please post them?
 
nocloud said:
Does anybody know the best way to check data integrity on Linux? In my situation, I need to keep approximately 12TB of data constantly online (4x3TB). The system will be JBOD since each 3TB disk holds static data that is backed up offline on another 3TB disk.

Is there a way in Linux to periodically scan all 12TB of data to check for data integrity? I would like to make sure that all bytes are still readable and if one file is unreadable, to give a warning message so I can delete the bad file and replace it with a copy from backup.

I'm concerned about sectors going bad and individual files within the 12TB slowly going bad and becoming inaccessible when they are requested.

Also, don't suggest things like XFS, etc. I need maximum compatibility, so Centos 6 with ext4.
Click to expand...

If you are using Linux then Linux-Mag has an article on Checksumming Files to Find Bit-Rot which looks interesting.

I am a command line newbie so will need to spend some more time figuring this out.
 
You must log in or register to reply here.
Back
Top