Virtual Drives and ZFS

A

avattz

n00b
Joined
Jun 25, 2012
Messages
26
Hi guys, I had a server running alongside my desktop workstation most of the time but over the past summer the heat output was unbearable. I'd also like to save on space and so I've decided to decommission my server and move the server hard drives into my desktop and run FreeBSD with VirtualBox to access my ZFS pools.

I've tried adding the physical hard drives to VirtualBox but it was a no go. My thought was to create VDI or VMDK taking up 95% of the server hard drives and just add them to VirtualBox. Now my question is, is it a good idea, does ZFS work well with "virtual drives", if bit rot happens on the physical drive, will ZFS be able to "repair" the virtual one?

I've read that its a bad idea to virtualize ZFS, but I'd prefer running a single computer nowadays.

Also I almost destroyed one of my zpools because Windows asked me if I wanted to initialize the test zpool drives, so having an NTFS drive with a huge VDI/VMDK should bypass this problem, with this though, I will have 6 hard drives in "My Computer" with no use, not sure if you can hide hard drives from users.

Edit: Basically what I'm asking is whats the best way I can tackle this problem, running ZFS under Windows.
 
Why not just put a low power CPU on your server ? I doubt the heat output is really significant even now, frankly. Unless you run BOINC projects on your CPUs and GPUs like me, then yes I feel the heat !
 
avattz said:
I've read that its a bad idea to virtualize ZFS, but I'd prefer running a single computer nowadays.

Edit: Basically what I'm asking is whats the best way I can tackle this problem, running ZFS under Windows.
Click to expand...
There are several reports of entire zpools getting corrupt under VirtualBox.

First of all, Windows is not that Enterprise stable, it might corrupt your data.

Second, VirtualBox is not stable, it has not the best reputation. If you want stable, you should use VMware for virtualization instead.

There are several levels of fail here. If you really want to run Windows, why use ZFS at all? ZFS is for the secure and safety concerned people. Combining it with Windows ruins the whole idea from ZFS.

You have at least three choices as I see it:
1) Use Windows 8 and Storage Spaces, which is a ZFS wannabe.

2) Use hardware raid instead. I would trust more on hardware raid + Windows, than trust ZFS + VirtualBox + Windows which HAS corrupt entire zpools. Hardware raid is safer than ZFS, is you insist on using Windows as host OS.

3) Virtualize with ESXi and run Windows in one VM, and OpenIndiana in another VM. Many people here do it, and there are great help from Gea_ on this.
 
Thanks for the responses guys, here is basically why I can't give up either machine.

Windows: because most, if not all, my applications run under Windows.
FreeBSD: because I can use ZFS, which has given me peace of mind even if its overkill just to keep static files.

I tried using the lowest output CPU I could find, which would be an Intel Celeron 1.6GHz which is rated for 65 watt TDP.

I have 8 disks in a Fractal Design Define R3, so most of the heat comes from the hard drives. I've thought of downgrading to 4 disks, but I found making a second computer pointless since my CM 690 II has 6 bays for hard drives.

It seems that virtualizing may put my data at risk, but at the same time I liked the idea of having a single computer since its almost always on anyway.

I was looking at snapRAID, but apparently the parity file can corrupt, and there is nothing that can rebuild it except the data itself, which could be corrupt too.


I do not mind and actually prefer to use mirror ZFS since its probably the most incredible thing invented (each hard drive can depend on the other to see if they are correct or wrong, I haven't found anything else that does the same thing).

You have at least three choices as I see it:
1) Use Windows 8 and Storage Spaces, which is a ZFS wannabe.

2) Use hardware raid instead. I would trust more on hardware raid + Windows, than trust ZFS + VirtualBox + Windows which HAS corrupt entire zpools. Hardware raid is safer than ZFS, is you insist on using Windows as host OS.

3) Virtualize with ESXi and run Windows in one VM, and OpenIndiana in another VM. Many people here do it, and there are great help from Gea_ on this.
Click to expand...

1) Thanks for the heads up, I'll look into Storage Spaces.

2) From what I've read, hardware RAID doesn't protect from bit rot, it even copies it over to the good side, I'm not sure about this though.

3) Not sure what ESXi is but will check it out.
 
I think your confusing cpu ratings. watt TDP is how much HEAT the cpu gives off, not how much power it draws. If the cpu stays in low power idle mode most of the time, you need to locate a cpu with that at the lowest, but it might scale up into an 80watt TDP when running full speed.

Though generally the atom cpus are best for this.
 
While your statement is basically true, a CPU will always output as much power as heat as it draws electrical power. The Sandy/Ivy Bridge Intel CPUs consume almost nothing in idle while proving significantly more on-demand performance as well as more features. I would prefer them over an Atom any day. I just built a router with a Celeron G460, it consumes about 17 watts in idle including a 7200rpm laptop drive.
 
Yes, it will draw and produce heat basically equally, BUT the MAX heat it produces has no bearing to the idle power used.

My atom system with 7200 laptopdrive draws 8watts.
My lenovo 530, with i7 quad, and ssd draws 15watts.

I personally haven't been able to get a non-atom system to sub 10watt range, except for arm systems.
 
avattz said:
1) Thanks for the heads up, I'll look into Storage Spaces.

2) From what I've read, hardware RAID doesn't protect from bit rot, it even copies it over to the good side, I'm not sure about this though.

3) Not sure what ESXi is but will check it out.
Click to expand...
1) I think it is another name for MS new filesystem called ReFS in Win8. ReFS uses Storage Spaces, which is some kind of raid. ReFS has some checksums but not complete checksum functionality.

2) Hw raid does not protect against bit rot, no. But chances it corrupts your whole raid is less than using VirtualBox and ZFS - I guess. If you are not concerned about loosing your whole raid, why are you concerned about bit rot?
Loose my whole raid? Sure, no problem. I dont need to protect against this.
Bit rot? Hell no, I dont want that! I need to protect against this.

3) ESXi is another virtualization technique used in Enterprise large fortune 500 companies. It is free to download and use if you have a small server. Here is a big how to
http://hardforum.com/showthread.php?t=1573272



Myself is running Solaris 11 and VirtualBox. Inside VirtualBox I have put Windows. Windows never gets raw acess to the zfs raid, but only via CIFS/NFS. This means Windows will not corrupt my zfs raid. Thus, the host OS is Solaris which is stable and use ZFS correctly. Then I fire up Windows in VirtualBox. Thus, Solaris still manages ZFS. If Windows crashes, it does not affect Solaris nor ZFS.

So you could run FreeBSD on your server, and then install VirtualBox ontop FreeBSD. And then fire up Windows. And FreeBSD serves Windows. This is a safe solution.
 
Although development has been start&stop, you can check out zfsguru - it has virtualbox integrated as a service.
 
So is this virtualbox integrated service a way of appeasing those storage users who want the benefits of a ZFS file system while maintaining the familiarity of a Windows environment (like WHS)?
 
brutalizer said:
1) ....
2) Hw raid does not protect against bit rot, no. But chances it corrupts your whole raid is less than using VirtualBox and ZFS - I guess. If you are not concerned about loosing your whole raid, why are you concerned about bit rot?
Loose my whole raid? Sure, no problem. I dont need to protect against this.
Bit rot? Hell no, I dont want that! I need to protect against this.

...
Click to expand...
try to clear up a myth where circulating for many years:
where do you get the information that HW raid can not protec/detectt bitrot?
how about RAID 6...
newer HW raid is more advanced than previous version
rule of thumb, bit rot can be happen anytime, either sofware or hardware level could detect.
Software solution is the cheapest and flexible, where mostly reside on the filesystem.
 
danswartz said:
Not sure what you mean by 'a windows environment'?
Click to expand...

As an alternative to the OP's TL;DR of running ZFS in Windows, running Windows on top of ZFS using VirtualBox. Thus, getting the benefits of ZFS in a (virtual) Windows environment.
 
try to clear up a myth where circulating for many years:
where do you get the information that HW raid can not protec/detectt bitrot?
how about RAID 6...
Click to expand...

I agree. raid6 does protect against bit rot provided you do not let 2 drives get kicked out of the array at the same time. On top of that weekly checks of the array should be performed to further reduce the chance of this happening.

Edit:
Yes I know raid6 will also not protect you if 2 or more members of the same stripe have bit rot however this scenario is will be highly unlikely.

newer HW raid is more advanced than previous version
Click to expand...

I do not think any of them have checksums on each member of a stripe so if the array gets degraded by 2 drives there is no way to detect or repair bit rot.
 
Last edited:
I no longer have my hard drives in RAID, they are all just paired up in mirrors, as if by horrible luck, both mirrors of one vdev fail, the entire RAID loses all the data. Now I only have pools composed of 2 disks in mirror.

I am not after performance in read or write, I just want to be able preserve data while being able to use it too. I like ZFS for its block-level checksum, and I even considered snapRAID because of this, but snapRAID works over the OS, which to me doesn't sound very robust.

I've been reading that you can "make" a computer into a router, so I might just combine router/server/nas with the smallest case with as many 3.5 drive bays (e.g. Node 304), but thats going to be a huge "router" lol. Heat output is what had me concerned recently, and I've checked out out i3 vs Atom articles, so I'll just suffer with the heat till 11nm processors come out.
 
drescherjm said:
I agree. raid6 does protect against bit rot provided you do not let 2 drives get kicked out of the array at the same time. On top of that weekly checks of the array should be performed to further reduce the chance of this happening.

Edit:
Yes I know raid6 will also not protect you if 2 or more members of the same stripe have bit rot however this scenario is will be highly unlikely.



I do not think any of them have checksums on each member of a stripe so if the array gets degraded by 2 drives there is no way to detect or repair bit rot.
Click to expand...

2 drive die instantly? :)..
everyone can say " XYZ will not protect, if 3 or more members....."
RAID is not a backup :)...

once 2 drive or more are failing instantly/concurrently ( base on RAID6 or RAIDZx), the system can hardly in an intact shape :).

bitrot happens in any RAID... as I know, where rarely happens when we have RAID6 or RAIDZx. some of us always say "RAID is Not a backup" where I totally agree.
unless a new breakthrough beyong RAID technology..

as I said, software raid gives us lower cost and flexibilty :).
on the other side, HW raid give us simplicity :D, for example , just remove raid card to a replacement server. boom.. all set ..

I try not to argue, just spread information that do not misleads software raid( especially zfs) vs hardware raid.
 
As said by others, RAID is not backup. avattz, you might want to review your strategy, use less live drives (with for example a single pool in RAIDZ2), thus reducing heat output, and using the remaining drives (and maybe more) to do offline backups.
 
Yes I know, I plan on mirroring the vdev mirrors and storing that copy somewhere else. I was wondering if I could attach and detach so I could scrub to update the backups.
 
That is how I back up my root pool. Plug in ext sata drive, attach to rpool disk, wait for resilver, detach, unplug...
 
I run Linux with ZFSonLinux as a host OS and virtualbox for my various guest OSes. I don't create giant VDI or VDMK files, but modest sized ones for booting and use the Virtualbox share folders to map to my directories on my ZFS mount.

If you don't like Linux/ZOL, you could do the same with OpenIndiana as a host also. I've also played around with making a Zvol and sharing it out via ISCSI and use a windows ISCSI initiatior to mount it. This would allow NTFS on a ZFS protected device.
 
Only complication is the very first time you need to partition&slice the backup drive so you can use the 's0' at the end of the disk name.
 
RAID is so much more trouble than it's worth these days. SSDs are so fast by themselves.
 
Billy_nnn said:
True enough, but the data doesn't survive drive failure very well! ;)
Click to expand...

Which is why you should have multiple backups of your important data on more than 1 drive, tape, cd / dvd, array or other backup media.
 
Last edited:
cantalup said:
try to clear up a myth where circulating for many years:
where do you get the information that HW raid can not protec/detectt bitrot?
how about RAID 6...
Click to expand...
Here are several research papers on hw raid and bit rot
http://hardforum.com/showthread.php?t=1638827&highlight=zfs+data+integrity+cern

RAID6 is unsafe. I have posted on RAID6 in the link above. I quoted a research paper on RAID6, saying something like
"In RAID6, if there is corruption then the RAID6 does not know for sure and guess instead. It is 95% chance the RAID6 will correct the corruption, and 5% chance of making it worse".

Read the link and study my posts for more information and see the original text.

The thing is, hardware raid does not detect data corruption. Hardware raid does XOR parity calculations, but that is not data integrity calculations to detect corruption.

Hardware raid has lots of problems, they are not built to detect nor correct data corruption. They repair disks if they crash.
http://en.wikipedia.org/wiki/RAID#Problems_with_RAID

More research papers on data integrity and some on hardware raid (by NetApp, for instance)
http://en.wikipedia.org/wiki/ZFS#Data_Integrity
 
brutalizer said:
Here are several research papers on hw raid and bit rot
http://hardforum.com/showthread.php?t=1638827&highlight=zfs+data+integrity+cern

RAID6 is unsafe. I have posted on RAID6 in the link above. I quoted a research paper on RAID6, saying something like
"In RAID6, if there is corruption then the RAID6 does not know for sure and guess instead. It is 95% chance the RAID6 will correct the corruption, and 5% chance of making it worse".

Read the link and study my posts for more information and see the original text.

The thing is, hardware raid does not detect data corruption. Hardware raid does XOR parity calculations, but that is not data integrity calculations to detect corruption.

Hardware raid has lots of problems, they are not built to detect nor correct data corruption. They repair disks if they crash.
http://en.wikipedia.org/wiki/RAID#Problems_with_RAID

More research papers on data integrity and some on hardware raid (by NetApp, for instance)
http://en.wikipedia.org/wiki/ZFS#Data_Integrity
Click to expand...

I will let you decide. nothing is safe..
reseach papers are on going discussion and discovery that can be accepted or denied....
The better way is start digging RAID technology and the emerging of Software RAID and many filesystems. everything is grey...

ZFS is not safe too.
as I replied to you on some threads, you should know why we have hardware RAID and software RAID.
I rarely take wikipedia 100% in my references..

software RAID or Hardware RAID have limitation,

RAID6 implementation is not identical between vendors , even mdadm has its own implementation.

if you are talking RAID, think about checksum/parity/cache/ and other technolgy.

been in Hadrware and software RAID. I always smile when around me debating Hardware versus Software. never ending discussion.
the questoin for me: what do you need to achieve, your budget and your environment. starting from there, you can pick among hardware/software raid

non filesystem or RAID can detect corruption from any form on the fly :).
Checksum can help to detect, but not a life saver...
Backup is your life saver!!!

until a newer technology to replace RAID. people would debate Hardware versus Software Raid..
 
cantalup said:
ZFS is not safe too.
Click to expand...
Of course nothing is 100% safe. Everything can fail. But the point is that ZFS is designed to detect and catch all sorts of data corruption (because ZFS uses checksums to see if any data has been altered). Hardware raid is not designed to do that (because there are no checksums). This shows that ZFS is safer, if we talk about protection against data corruption.


I rarely take wikipedia 100% in my references..
Click to expand...
Me neither. But on the wikipedia link, there are research papers I was refering to. And I trust comp sci researchers more than wikipedia. Thus, on the wikipedia link, there are several research papers. Never mind the wikipedia text, but read the research papers.


RAID6 implementation is not identical between vendors , even mdadm has its own implementation.
Click to expand...
The RAID-6 paper I was refering to, talked about Reed Solomon codes that most (all?) RAID-6 are using for parity calculations. They are implemented differently, but they are (all?) using the same mechanism, which is flawed. R-S codes are flawed, which the mathematical research paper talks about.


if you are talking RAID, think about checksum/parity/cache/ and other technolgy.
Click to expand...
There is a big difference between checksum calculations and parity calculations. Hardware raid does not have checksums to detect data corruption. If you claim they do, please show me a link to a hardware raid that does.

As researchers write in a paper:
"A real life study of 1.5 million HDDs in the NetApp database found that on average 1 in 90 SATA drives will have silent corruption which is not caught by hardware RAID verification process; for a RAID-5 system that works out to one undetected error for every 67 TB of data read"

Thus, hardware raid are notoriuosly bad at detecting data corruption, because they are not designed to do that. How can you correct a corruption, that you can not detect?

Lastly: hardware raid does not have END-TO-END CHECKSUMS. End of this discussion.
 
I'm currently doing parity calculations of all my files, both on the main drives and on their backups, and I'm finding lots of corruption, especially on big video files. I don't know what is the cause, for starters I only own desktop machines so no ECC memory, but it's far more than I would expect, I must be doing something wrong.
 
How are you doing "parity" calcutlations? Hardware raid is doing parity calculations when repairing disks. Are you using hardware raid?
 
Sorry I was tired I guess, I'm doing file checksums (CRC32). I don't use any kind of RAID or ZFS, I'm considering it, but my priority is now to check my data.
 
danswartz said:
That is how I back up my root pool. Plug in ext sata drive, attach to rpool disk, wait for resilver, detach, unplug...
Click to expand...
But for booting from that drive,you need to partition/slice properly it,SMI label ,resilver ,installgrub ;)
 
Yes, I know. I thought I had mentioned that earlier :) I omitted it here because it's not required after the first time.
 
brutalizer said:
Of course nothing is 100% safe. Everything can fail. But the point is that ZFS is designed to detect and catch all sorts of data corruption (because ZFS uses checksums to see if any data has been altered). Hardware raid is not designed to do that (because there are no checksums). This shows that ZFS is safer, if we talk about protection against data corruption.



Me neither. But on the wikipedia link, there are research papers I was refering to. And I trust comp sci researchers more than wikipedia. Thus, on the wikipedia link, there are several research papers. Never mind the wikipedia text, but read the research papers.



The RAID-6 paper I was refering to, talked about Reed Solomon codes that most (all?) RAID-6 are using for parity calculations. They are implemented differently, but they are (all?) using the same mechanism, which is flawed. R-S codes are flawed, which the mathematical research paper talks about.



There is a big difference between checksum calculations and parity calculations. Hardware raid does not have checksums to detect data corruption. If you claim they do, please show me a link to a hardware raid that does.

As researchers write in a paper:
"A real life study of 1.5 million HDDs in the NetApp database found that on average 1 in 90 SATA drives will have silent corruption which is not caught by hardware RAID verification process; for a RAID-5 system that works out to one undetected error for every 67 TB of data read"

Thus, hardware raid are notoriuosly bad at detecting data corruption, because they are not designed to do that. How can you correct a corruption, that you can not detect?

Lastly: hardware raid does not have END-TO-END CHECKSUMS. End of this discussion.
Click to expand...

this discussion can goes along in time :D
if you do not 100% on wiki, please post research papers link directly..nothing to worry about wikipedia.

HW vs SW RAID...
I am reluctant to bring something on this thread where would goes OOT

"A real life study of 1.5 million HDDs in the NetApp " by whom hehehe
there are some agrees and disagrees

let you decision works.


check on other research papers too, for example filesystem( modern filesystem) research papers, they do not have conclusion, unless we can have a breakthrough that we have now...
RAID could be replaced in the long future.

nothing is perfect, everything has flaws..

--------
when speaking data corruption where can be happened during transferirng, on the fly or off the fly or others
research papers always hiding something where they can publish another papers in the future, Had been there writing for my profs during my study in CS 10 years ago...
Learn alot from my profs at that time on how to Not publish much information in papers due on continuation of Researches.( specially to attract funding).

I read "Tanenbaum, Modern Operating Systems" that discusses modern filesystem :). just for fun :),

ok end of discussion :),
 
cantalup said:
this discussion can goes along in time :D
if you do not 100% on wiki, please post research papers link directly..
Click to expand...
Ok, I can do that. The thing is, I just had this same discussion earlier here. I wrote lots and lots, and linked and quoted research papers. It took much time, but I can do it here. Again.

ok end of discussion :),
Click to expand...
No, not end of discussion. I will post links again on this so you understand that hardware raid is totally unsafe with respect to data corruption.

HW vs SW RAID...
I am reluctant to bring something on this thread where would goes OOT

"A real life study of 1.5 million HDDs in the NetApp " by whom hehehe
there are some agrees and disagrees
Click to expand...
Here is a researcher at NetApp who wrote some of the research papers I have been linking to. Have you heard about NetApp? One of the biggest Enterprise storage companies with storage servers costing millions of USD. NetApp is one of the best and most trusted in Enterprise storage where price is no concern.
http://www.netapp.com/uk/company/leadership/advanced-technology/lakshmi_n_bairavasundaram.html

A team of researchers (including this guy) writes "A real life study of 1.5 million HDDs in the NetApp database" here. They examined 1.5 million hard disks that had been installed at 50.000 Enterprise customers and found out that "1 in 90 SATA drives will have silent corruption which is not caught by hardware RAID verification process; for a RAID-5 system that works out to one undetected error for every 67 TB of data read". I mean, when you read research papers, they seldom include lots of material, much of computer science research is only theoretical. That these guys examines 1.5 million HDDs at the largest storage company, is very trustworthy and credible. Dont you agree?
http://media.netapp.com/documents/camready.pdf


Further, in the ZFS wikipedia link, it says
"For example, a modern Enterprise SAS disk specification estimates this fraction to be one uncorrected error in every 1016 bits".
Every disk, gets lots and lots of errors on the fly during usage. Most of the errors are corrected on the fly by error correcting codes. But some errors are not correctable. For instance, the error correcting codes might be able to correct 1 bit errors and detect 2 bit errors. But very seldom, 3 bit errors will occur, maybe if you hit your computer during read. And these codes maybe not can correct 3 bit errors, nor 4 bit errors. etc. Have you studied error correcting codes in Abstract Algebra? If you have, you know they have limitations.

Some of these errors are not even detectable. Let me repeat, some errors are not detectable. If you study the specification sheet for an Enterprise SAS disk, you will see typically see "One bit of 10^16 are not correctable": If you study the spec sheet of an Fibre Channel disk (those are VERY high end and expensive) it says on page 17:
--Unrecovered Data Less than 1 sector in 10^15 bits transferred
--Miscorrected Data Less than 1 sector in 10^21 bits transferred
http://www.seagate.com/staticfiles/support/disc/manuals/enterprise/cheetah/10K.7/FC/100260916d.pdf
Some errors are even MISCORRECTED. And other errors are not even detectable. At page 18 they speak about INTERFACE ERRORS. The Interface might even show errors. You know, the cable. The connection. etc etc. Here is information about Interface errors:
http://en.wikipedia.org/wiki/Soft_error

And if you hit the disk, it will be affected and might show errors. If you play loud music close to the disk, it can show errors. If you even scream at the disk, it can show errors:
http://www.youtube.com/watch?v=tDacjrSCeq4

And then we have cosmic radiation. For instance, "Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month"
http://www.scientificamerican.com/article.cfm?id=solar-storms-fast-facts

"Intel Corporation has proposed a cosmic ray detector that could be integrated into future high-density microprocessors, allowing the processor to repeat the last command following a cosmic-ray event."
http://news.bbc.co.uk/2/hi/technology/7335322.stm



Do you now understand that there might be all kind of different errors? Do you understand that some errors are not correctable? Not even detectable? For instance, if there is an error in the cable (interface) how can the hard disk correct that error? The HDD does not know it is an error, it gets "01110101011...1" but one of them got corrupted by the cable, and the HDD has no way of knowing. When the HDD reads data, it can detect and correct errors, but if the HDD gets wrong data from the start, the HDD will just save the data without knowing if the data is correct or not.

Sometimes even the router or the switch will be faulty. And the HDD has no way of detecting a faulty switch. Do you understand that there are many error sources, and that the hardware raid has no way of correcting nor detecting all errors? Do you agree on this?


check on other research papers too, for example filesystem( modern filesystem) research papers, they do not have conclusion, unless we can have a breakthrough that we have now...
Click to expand...
Here is research that shows that filesystems such as Linux EXT, JFS, XFS, NTFS, ReiserFS, etc are all unsafe with respect to data corruption:
http://pages.cs.wisc.edu/~vijayan/vijayan-thesis.pdf
All these filesystems does not correct all types of data corruption, and they can not detect all types of data corruption. He writes that they show inconsistencies in error detection and that he tries to inject artificial errors that they can not detect, nor correct.

Are you know convincend that hardware raid is not safe? If the disk might miscorrect errors, the disk will not notice it. And the hardware raid card will not notice, because the disk will not tell the raid card. Even if the data is correct from RAM down to raid card, the interface to the disk might corrupt the data, or the disk miswrites data. Some data is not even written, but the disk believes he has written data - these are called Ghost Writes. The writes were in the air.




Regarding ZFS, it is a totally different thing. It has end-to-end checksums which changes everything. ZFS will compare what is in RAM to what is written down on disk. If the cable corrupts data, ZFS will notice it. If the disk miscorrects data, it will notice it. ZFS compares the data from the beginning to the end on the chain. No one else does it. Therefore, if the disk makes an error, but does not know it, ZFS will detect it. And correct it. Hardware raid has no end-to-end checksums. You must have control of the whole chain to get end-to-end checksums, from RAM down to disk. Only a monolithic filesystem such as ZFS can do it. If you have several layers, such as raid, then you break up the chain and you loose end-to-end integrity.

Here is an example when ZFS detects there is an faulty Fibre Channel switch. Because ZFS will compare what is written on the storage SAN server, with data in RAM on the client, ZFS will notice the switch is faulty. No one else noticed it, ZFS was the first to notice it.
http://jforonda.blogspot.se/2007/01/faulty-fc-port-meets-zfs.html
"there was no mirroring or protection at the server level, we had delegated that function to the DMX3500

I've seen this kind of "trust" in some sites. But that turned out to be a bad decision:

As it turns out our trusted SAN was silently corrupting data due to a bad/flaky FC port in the switch. DMX3500 faithfully wrote the bad data and returned normal ACKs back to the server, thus all our servers reported no storage problems.
...
ZFS was the first one to pick up on the silent corruption"



Ok? Only a monolithic filesystem that controls everything can have end-to-end checksum. If you break it up into parts, it is not possible.

Have you played a whisper game? One person whispers a word to another person, that whispers the word further, etc. Then we compare words from the first person to the last, they always differ. You need to compare the beginning and the end, to see if they are the same. End-to-end. ZFS.

The thing is, I have written all this stuff again in one of the links I posted. And now I spent an hour to write the same things AGAIN. This sucks. Here I write exactly the same thing as in this post, but much more and many more links:
http://hardforum.com/showthread.php?t=1638827&highlight=zfs+data+integrity+cern

Please read it, so I dont have to write exactly the same stuff here again. It takes time.
 
Thanks for the additional information brutalizer. End-to-end checksum is the primary reason why I use ZFS, but I didn't even think of errors induced by outside-of-the-computer-case events like cosmic radiation.

I've semi rebuilt my server computer by removing empty drives (I'll add them as I run out of space) and by removing a SATA controller and two HP server dual ethernet cards, which were the hottest things in the case (I couldn't touch the heatsinks for more than 2 seconds). The case runs much cooler now with just 3 drives and an SSD.

I've also come across more information regarding labels and ZFS (since I removed my SATA controller, I "lost" my drives to FreeBSD) and apparently labels use the last sector(s) of a drive without ZFS knowing, ZFS think it owns the entire drive, and may lead to corruption. Any way to avoid disaster? I know its just the last sector of the drive, but you know me, I prefer not to risk my data.
 
Sorry, I dont know about FreeBSD and labeling it does. Ask in the huge Opensolaris derived Napp-it thread? Or are there any FreeBSD users here?
 
Search like 'glabel gpt zfs' .... either don't use glabel or use glabel and put zfs on a partition instead of entire disk, or just use gpt labels. If you google you'll find more about it.
 
You must log in or register to reply here.
Back
Top