MDADM to ZFS build

Ciamician

n00b
Joined
Sep 22, 2013
Messages
20
I'm planning on upgrading my current NAS to ZFS. You can find my current build, aswell as some of the specs of my future build here.

I'm thinking now would be the ideal time to switch to ZFS, since I have a 4TB external drive where I can transfer all my current data which happpens to be around 4TB (3x2TB in RAID5 via mdadm). I'm still pretty clueless when it comes to the mobo, cpu & controller and could definately use some advice in that departement.

I'm also thinking about only buying one extra 2 TB drive for the moment, and invest in the core of my new ZFS build and when the time is right I can purchase an additonal controller and more hard drives.

Expanding your array in ZFS however doesn't seem to be as painless as it was in mdadm. Would it be wise to operate like this, or should I wait it out a little bit longer so I can save up a bit more and can all my drives at once?

Controller-wise, I've been looking at a lot of cards. IBM M1015's seem to be really popular, but I don't really like the placement of the 2 ports since they are pointing upwards and would result into poor cable management. I mentioned a few alternatives in the excel doc (at the top), but I'm not planning on spending too much on a controller since its only purpose is adding more slots (from a ZFS pov). I do however want a controller which has SAS 8087 slots, as I'm planning on connecting them via an internal 8087-to-8087 cable to improve cable management.

Thanks in advance :)
 
Assuming that most of the data will be media files (write the files once or seldom, read the files back frequently), then your best bet for a fileserver that will be slowly expanded is to use snapshot RAID. Either SnapRAID or FlexRAID.
 
There is a card based on the same chipset which has end ports rather than top ports, which is quite suitable for ZFS - LSI 9201-8i. There's a couple of eBay sellers with them - we use 5 or 6 and they're quite good (identical to a M1015 in practice).
 
Assuming that most of the data will be media files (write the files once or seldom, read the files back frequently), then your best bet for a fileserver that will be slowly expanded is to use snapshot RAID. Either SnapRAID or FlexRAID.
I agree with this. If you are mostly doing media, it does not matter if you get some slightly corrupted data, because it does not matter if you get a white pixel instead of red. If you are doing media, Snapraid would be good. The best of two worlds would be to run snapraid ontop ZFS of course:
http://forums.servethehome.com/sola...-zfs-raid-snapraid-media-server-napp-box.html
 
i disagree. the worst thing in the world is having to re-encode a movie because a SLIGHT bit of corruption knocked the audio out of sync. im not sure i would use snapraid or flexraid for anything. other then gaining a small bit of parity, it takes all of the bad things about using multiple single disks and spreads them across multiple disks.
 
Not sure what's so bad about FlexRAID. It has good enough bitrot/corruption detection. All the files are checksummed so if you get data corruption then it will tell you and you have the opportunity to restore the file via parity.

Yes, it's a manual process to fix bitrot/corruption errors unlike the automatic process in ZFS, but It's good enough for a media server and in my experience you don't run tinto bitrot or data corruption all that often.
 
My server will have media files, but that's not all. ZFS seems like an excellent choice, considering some of the features it has, so lets keep it aside for now :).

I would still llike to get a couple more answers surrounding my plan to first purchase the core of my server and leave expanding the storage capacity for later, and some advice on the hardware (mobo, cpu & controller).
 
I've read enough horror stories of a disk failing during expansion that I just bite the bullet, have a backup (or two!), and burn it down and rebuild when I need to "expand" an array.

I'm pretty turned off to hardware RAID controllers other than LSI and Areca. I find linux software parity RAID(5/6) to be acceptable, and generally have had really good luck with ZFS so far.

Instead of RAID5, you might consider having a ZFS share with single disks that is extended, and a second full mirror in another machine.

Expanding a single volume is a piece of cake if there is no parity calculations involved, and the whole time you have a full backup, which is still all check-summed.
 
im currently building a ZFS / VM box. (it will be Proxmox for VM, though this forum seems to like ESXi)

the affor mentioned case is the base of the build. internal is a Gigabyte 990fxa-ud5 wtih a 6 core AMD CPU and 16gb of PC3-ECC ram.

also 2x dell H310 HBA's they are in DELL IR mode but i might flash them to DELL IT mode. i have no desire to crossflash to LSI firmware. and either of the DELL firmwares support passthrough of the drives.

also i will be using a dual port intel based gig nic, and probably set up the on mainbord nic as a management port.

the video card and other features are all but irrelevant in this build. if you were hardcore you could use the com port and actually be truly headless even during setup.

i am an AMD guy, intel is overated and overpriced. (ignore the intel nic i am using, it was cheap and they are good for VM builds.)

my build will also be mostly media but enough other stuff to make real ZFS a worth while investment.

i come from hardware RAID and prefer adaptec, but i have used others too. as well as MDADM but i never liked MDADM for large arrays. mostly i would use MDADM for an array for the host server OS and then build out a hardware array for storage, so this ZFS thing is a learning experience for me too.
 
Seems like Avoton CPU's should be coming around november. Perhaps I should wait a little longer?
 
if i was wanting to do a low power build i would use an AMD APU. there have been a few times i have been forced to use things that have the 'Atom" brand on them and it has scared me from ever even allowing anything with 'intel' and 'low power' to ever be purchased.

if i were looking for ultra-low power i would use something non-x86. a beagle board or rasbery-pi or one of the dozen other Arm based SoC boards.

i have used tons of intel products, most of the places i have worked at were strictly intel. i will never understand why, bang for the buck is 99.9% of the time in AMD side. and seriously, out of everyone i know, NO ONE is on the bleeding edge, so save a few $ and get an AMD chip.
 
if i was wanting to do a low power build i would use an AMD APU. there have been a few times i have been forced to use things that have the 'Atom" brand on them and it has scared me from ever even allowing anything with 'intel' and 'low power' to ever be purchased.

if i were looking for ultra-low power i would use something non-x86. a beagle board or rasbery-pi or one of the dozen other Arm based SoC boards.

i have used tons of intel products, most of the places i have worked at were strictly intel. i will never understand why, bang for the buck is 99.9% of the time in AMD side. and seriously, out of everyone i know, NO ONE is on the bleeding edge, so save a few $ and get an AMD chip.

Ok. Any advice on the chipset? I've always used intel, but from what I've read.. ZFS relies more on RAM then it does on CPU power.
 
I understand the aversion to the Atom name above, but the Avoton chips are a whole different beast coming from collaboration with the Xeon team. If you're wanting low power consumption they'd be a great choice, and also have ECC support. Don't expect pricing to be like the previous Atom chips though.
I'd recommend looking at SuperMicro's motherboards for a build like this.
Are you planning on doing anything else on this server or is it purely for storage?
 
Mostly storage yes, but it also serves as a webserver, streaming to multiple hosts in my network (also via DLNA), SABNZBd,...

I'm still thinking about switching to a 3U/4U Norco case. Would it be silly to do so considering they have 16 slots and will only be able to fill up 4 for now?
I'm planning on gradually filling more slots as my storage needs grow.
 
Last edited:
if you would like to jump on ZFS and do not need many funky(complete) feature of ZFS,

moving from mdadm to ZFS on Linux (ZoL) is painless, since you can apply your linux knowledge directly to ZOL with additional ZFS configuration and knowledges.

you do not need to flash to IT firmware
I use Dellizer HBA card H2XX an H3XX series on ZoL(running on Centos 6.3), those cards will configure "unconfigured" drives as JBOD, and the good thing is Linux kernel drivers for those cards are stable :D.
if you need to flash to IT firmware, flashing to dellizer HBA card( IT firmware) is not much scary to do( Not cross-flashing to other non Dellizer firmware).

my slogan is "KIS(keep it simple)".

the motherboard (reused) is Asus (ecc supported) with AMD low power CPU with 8G DDR2 unbuff ecc RAM ( pretty old system compared as today).
 
Mostly storage yes, but it also serves as a webserver, streaming to multiple hosts in my network (also via DLNA), SABNZBd,...

I'm still thinking about switching to a 3U/4U Norco case. Would it be silly to do so considering they have 16 slots and will only be able to fill up 4 for now?
I'm planning on gradually filling more slots as my storage needs grow.

That much IO shouldn't be a problem for even say, 4GB of ram. just an FYI.
 
That much IO shouldn't be a problem for even say, 4GB of ram. just an FYI.

I'm talking about hard disks, not RAM slots :D.

Planning on getting 8GB ECC, but I can get my hands on 16 GB ECC for around $100, still thinking about it.
 
i like to stay with the socket CPU's though it is not a requirement.

i would look at a gigabyte 990fxa-ud3. it supports 32gb of ecc ram and has 4 pcie slots. and gigabyte has better support for sas RAID, and HBA cards then ASUS.

this would work nicely especially if you are looking at a full 3u or 4u case. it plays perfectly into the expandability of such system. you can plop in a dual core low power cpu now, and run off of the onboard sata ports for now. and start with a small amount of memory for now. and add components as you need them.

plus virtualization is supported so VM server with esxi or proxmox can also be rolled out as needed.
 
I hadn't seen this mentioned (although I sort of skimmed over stuff so I may have missed it) but there are only two ways to expand a pool once it is established.

The first is to swap out each disk with a larger one. e.g. You have 3 500 GB disks in a raid z1 and you replace each 500 GB disk with a 2 TB disk and let it rebuild onto each disk.

The other is similar to adding a raid 0 on top of it all. You have 3 500 GB disks in a Z1. You purchase 3 2 TB disks and create another Z1. You can then add the secondary 3x2TB Z1 to the 3x500GB Z1 to create the equivalent of a raid50.

The two ways you cannot expand is via adding addtional disks to an already established 'group' as well as changing the raid type. You cannot turn a 3disk Z1 into a 4 disk Z1. You also cannot turn a 3disk Z1 into a 4disk Z2. If you wanted to do either of these two options, you would need to backup your pool, nuke the drives, and rebuild them under the new configuration.

You do have to be careful about option 2 as ZFS will not stop you from making a stupid mistake. It will allow you to add a single, non raided harddrive to your pool which would give you a single point of failure for the entire pool. Be careful!
 
Not sure what's so bad about FlexRAID. It has good enough bitrot/corruption detection. All the files are checksummed so if you get data corruption then it will tell you and you have the opportunity to restore the file via parity.

Yes, it's a manual process to fix bitrot/corruption errors unlike the automatic process in ZFS, but It's good enough for a media server and in my experience you don't run tinto bitrot or data corruption all that often.
For large data, CERN, Amazon, etc - makes it very clear that just adding checksums does not make a storage solution safe. Just read their studies. Just because FlexRAID has some checksums does not mean it is safe.
 
For large data, CERN, Amazon, etc - makes it very clear that just adding checksums does not make a storage solution safe. Just read their studies. Just because FlexRAID has some checksums does not mean it is safe.

Neither does ZFS mean the data is safe, unless the data was originally written to ZFS and never copied out of ZFS.

For some applications, SnapRAID or FlexRAID's methods of managing checksums are actually better than ZFS. For example, you can copy your data around, even to different filesystems, and SnapRAID and FlexRAID will still be able to verify the checksums, since that information is maintained independent of the filesystem.
 
Neither does ZFS mean the data is safe, unless the data was originally written to ZFS and never copied out of ZFS.
I dont follow your logic. ZFS is not safe, if the data is not using ZFS? It is like, a vault is not safe, if you take out your money from the vault.

When ZFS gets data, the data is safe. ZFS makes guarantees that data is safe as long as ZFS handles the data. If data leaves ZFS, then ZFS can not make guarantees anymore, and why would it?


For some applications, SnapRAID or FlexRAID's methods of managing checksums are actually better than ZFS. For example, you can copy your data around, even to different filesystems, and SnapRAID and FlexRAID will still be able to verify the checksums, since that information is maintained independent of the filesystem.
As CERN and Amazon and numerours others show in studies: adding checksums is not enough. There are research on this. There also at least two research papers on the safety of ZFS, which I can show you. Can you show us research on the safety of SnapRAID or FlexRAID? On the other hand, I can show you research that NTFS is not safe, which SnapRAID / FlexRAID relies on. I would prefer to read research papers SnapRAID et al - can you show us such research papers?
 
I know you always say checksumming files is not safe brutalizer, but can you actually explain why it's not safe? Can you provide a real world example where it would not work in MY case and in my personal usage? I really don't understand why It's not a reliable way to detect bit rot for the way I store and use files.

If I have a file and I have that file's checksum stored somewhere, in what scenario will it not show me that something has changed? Because that's all I need to know, if a file has somehow changed unexpectedly.

If a bit in the file changes, then next time I run a checksum it certainly wont match. Or if a bit in the checksum somehow got corrupted, it also wouldn't match. In either case, it's going to tell me that something is wrong. First thing I will do is run the check again because it could have been a data read error. But If I get the same result again then I can be sure it wasn't a rear error. That's all I care about, because then I will go and restore that file from backup. Then I will check the restored file against the checksum. If it now matches then I can be sure I have the original uncorrupted file. If it still doesn't match, then the checksum is what must have got corrupted. This is why I keep a backup of the checksums as well.

I really do not see the problem.

I think those studies that you are talking about show that checksums are not enough for a self-healing array, and that is probably completely true. But what you don't seem to understand is that I manually check and manually heal/repair/replace my data myself. So in a manual case, can you still say checksums are not enough? Checksums are a simple and reliable way to determine if a file has changed.



I can explain my setup if that helps.

Every week I compute and compare all the MD5 checksums for all the files on my server. I store these checksums on another disk as well as on a separate backup system.

If the log shows that a checksum did not match, then I re-run the checksum on that file to eliminate the possibility of a read error or other error that caused the file to change between the disk and the software doing the checksum.

If it still doesn't match, then I go to my backups (my backups are created via deduplication and keep every version of a file, so if a bit were to flip, my backup would store that version too.)

So I go to my backups and grab the latest copy of the file and compare that to the checksum. If it doesn't match, then I work backwards in file versions to find the one before the corruption that matches.
 
Last edited:
There is no problem for home usage, only for scientific setups where an absolute guarantee that not a single bit is corrupted is needed to not invalidate the dataset or the resulting conclusions.

I have run weekly checksum verifications from calculated backup checksums of 8TB data (in the beginning), now 15TB on a 16 -> 28TB ext4/mdadm RAID6 array with montly scrubbing for a year now, and I have had 0 checksum errors in that period (my script sends me a warning by mail if this happens too). In the same period I have had to exchange 6 of the 8 crappy Seagate 2TB LP disks because of either complete failure or SMART failures. Even with an extremely high failure rate on the hardware with lots of detected SMART errors not a single bit have been flipped in the files on the mdadm array.

I would say that RAID6 with regular scrubbing and additional automated checksum verifications just to feel better is as good as it gets for 99% of all usage scenarios.

Not to say that checksumming filesystem shouldn't be used, as ZFS, BTRFS (any more?) should be used if it is possible, but you are probably never going to have checksum related issues in real life on RAID6 + scrubbing on a home setup, or even in large SAN-setups.
 
Yeah, that's how I feel.

I never questioned which storage system I would use at work. I would obviously choose ZFS.

But having had about 10TB of data already existing before I got serious about making a storage server and protecting my data, there was no economical way I could get that data into ZFS. And it's just not economical for a home user to grow the array as well. I like to grow my array 1 disk at a time. ZFS is just not flexible enough for this home user. And you can get more than enough reliability and protection other ways which do allow you to be flixible. So I use those systems. It's as simple as that.
 
Checksum battles aside..., could someone provide me with a bit more advice on the hardware I should choose to build my ZFS server? :)
 
It is not difficult. Think about it.
I studied lot of logic at university. I am a mathematician. I dont follow your logic. You say something like: if ZFS gets corrupted data, then ZFS is not safe. I dont understand what you mean. When ZFS gets the data it will be intact. If the data leaves ZFS, the data might not be intact, but that is not a problem for ZFS because ZFS kept its promise: to keep the data intact. You say: if data is copied off ZFS, then data might get corrupted, therefore FlexRAID is safer. But if data has been copied off ZFS, and gets corrupted afterwards - it is not ZFS fault. I dont understand your logic.

Could you please show us research on Flexraid or Snapraid? You keep saying that they are safe, how do you know? I can you show you research papers on ZFS and data integrity if you want to read. But how can you claim Flex/Snapraid are safe, without even basing your assumption on some evidence? Sure, they might be _safer_ than a non checksummed solution, but that does not make them safe. Unless you can present some research on this. Can you? On what evidence do you base your claim? Research? Or is it an opinion?



I know you always say checksumming files is not safe brutalizer, but can you actually explain why it's not safe?
No, I can not explain it, but can only cite other researchers on this. CERN concludes in a study that it is not enough to add checksums to make a storage solution safe. Do you want to read CERN's study?
 
Mobo, CPU, and RAM (at least 16GB) with ECC supported. WD Red or SG Nas hard drives. Seasonic or Corsair PSU.
What else, guys?
 
This is the problem I have with your whole advocation for ZFS to EVERY user then. You keep basing it on a study that you don't fully understand the scope of. You just use the fact that they reach "some" conclusion that has "something" to do with checksums not being enough alone.

But enough for what? That is my question.

I will ask you this.

If you don't actually know what the scope and the use case of the paper includes, then how can you with confidence say that for other people's use cases (or at least everyone that you advocate ZFS use to and claim that it's better for) such as the way I use my server, that checksums are not enough for those use cases.

If for example that CERN paper made the conclusion that checksums are not enough alone for a self-healing (from bitrot) array then that is great. I have zero doubts about that. But you are advocating ZFS to a user (me) and I do not care to have a self-healing array. I heal my array manually myself.

There is no direct evidence then that checksums along with manual human intervention might in fact be enough to prevent and recover from bit rot. Certainly the burden of proof to produce the reason my solution is no safe is on the advocate for ZFS.

I have come up with my own data protection solution that uses checksums, parity, a full deduplication backup with versioning, combined with human intervention to make the recovery and "healing" decisions. Since the CERN paper does include, nor test, nor compare ZFS to my custom solution, then you can't just use the CERN paper alone to say that my solution is not also as safe.

Of course I cannot say that my solution is safe without some study being done on it. But if I were to do a study then you would claim that I am not a reliable source. So how can I win? I can't

Surely you can see where I am coming from? You made it seem like the existence of the CERN paper automatically mean't my solution is not safe, when there is no way scientifically that it can make any such conclusion about my solution. Maybe you didn't intend to make it come across that way.


In order to really use the CERN paper to know how my solution compares, you would need much more than the conclusion alone (which is all you have ever quoted). One would need to read the paper in it's entirety and understand it all and all of it implications. Only then can you see if the weakness of checksums alone that they uncovered are taken care of by another solution (like my custom solution), or if they are still weaknesses that are subject to error given all the steps that that other solution takes.

ZFS uses checksums as well as other techniques to ensure data integrety. My personal solution uses checksums as well as other techniques to ensure data integrity. Saying that checksums alone are not enough to ensure data integrity is correct, but neither ZFS nor my solution use checksums alone to determine data integrity.

The important part of the paper is the "Because" part. "Adding checksums to a filesystem is not enough to ensure data integrity becasue..." And then we would need to see if the rest of my solution satisfies and fixes the because part or not.
 
Last edited:
This is the problem I have with your whole advocation for ZFS to EVERY user then. You keep basing it on a study that you don't fully understand the scope of. You just use the fact that they reach "some" conclusion that has "something" to do with checksums not being enough alone.

But enough for what? That is my question.
I am quoting researchers on this. I can not explain everything the researchers does. Can you derive Einsteins equations, for instance the famous E=mc^2? If not, why do you refer to it? My point is, if researchers say something, I dont really feel that I or you, must be able to explain their research, in order to quote researchers?

There are lot of research from NetApp, CERN, Amazon, etc - all big players that show that it is very difficult to protect against data corruption. Amazon has added checksums in many places, in many layers, and still they do get data corruption. Amazon say, it does not suffice to use checksums in RAM (ECC RAM), then they see errors on the storage. etc. If it was as simple to add a checksum as JoeComp says, then why are the big players having so much problems with data corruption? I am not able to explain all data corruption problems there are, but I know that all big players have problems - so why dont all of them migrate to SnapRAID? It is common sense. All researchers and big storage companies talk about all the data corruption problems they have - and then a guy makes SnapRAID and JoeComp claims it is safe? Does that sound probable to you? That a home brewn product claims to fix something that others have poured lot of brain power and billions(?) of USD into. We have at least two choices:
1) SnapRAID is safe. The guy has alone, succeeded in something everyone considers extremely difficult, and researchers and engineers at NetApp and other big storage companies, have big problems with.
2) SnapRAID is not safe. The guy is wrong and has not out brained all researchers and engineers on this subject.
What is most probable? Common sense? If something sounds too be good to be true, it is not true.

Amazon engineer explains
http://perspectives.mvdirona.com/20...ErrorsCorrectionsTrustOfDependentSystems.aspx

"Every couple of weeks I get questions along the lines of “should I checksum application files, given that the disk already has error correction?” or “given that TCP/IP has error correction on every communications packet, why do I need to have application level network error detection?” Another frequent question is “non-ECC mother boards are much cheaper -- do we really need ECC on memory?” The answer is always yes. At scale, error detection and correction at lower levels fails to correct or even detect some problems. Software stacks above introduce errors. Hardware introduces more errors. Firmware introduces errors. Errors creep in everywhere and absolutely nobody and nothing can be trusted.

Over the years, each time I have had an opportunity to see the impact of adding a new layer of error detection, the result has been the same. It fires fast and it fires frequently. In each of these cases, I predicted we would find issues at scale. But, even starting from that perspective, each time I was amazed at the frequency the error correction code fired. "

I dont mind if people says that a checksummed version is safer than a non checksummed, but to say it is safe is plain weird, in worst case you are fooling people. JoeComp claims SnapRAID is even safer than ZFS. I really wonder on what evidence does he claim this? I know every big player has big problems with data corruption, so is the SnapRAID developer totally wrong when he claims it is safe? Why dont everyone buy his company and migrate to Snapraid? Does it make sense he is better than all researchers and engineers? What are his credentials? Is he a former NetApp engineer? Or a self ttaught hacker who never has seen a storage hall? Is SnapRAID built from the ground up to combat data corruption, or is it an afterthought by adding some checksums?
 
...and then a guy makes SnapRAID and JoeComp claims it is safe?

You are making assumptions and claiming I said something that I did not say.

I never said "SnapRAID is safe". In fact, I will point out that data corruption is quite possible with SnapRAID or FlexRAID. For example, if the data is corrupt before the first checksum is computed, then it is impossible for SR to guarantee the integrity of the data.

Note that this problem applies equally to ZFS -- if the data is generated, for example, on a system with NTFS (perhaps ripping a disc on a Windows machine) and then the data is copied to a ZFS fileserver, then ZFS cannot guarantee the integrity of the data (it could, for example, have been corrupted in the copy operation from NTFS to ZFS).

In fact, this problem is worse for ZFS than it is for SnapRAID or FlexRAID. Since SR and FR maintain checksums on the data independent of the filesystem, after the checksum has first been computed, you can then copy the data among different filesystems, even filesystems without built-in checksums, and SR and FR can guarantee the integrity of the data starting from the time the checksums were first computed. ZFS, by itself, can never guarantee the integrity of data that has been copied to another filesystem, and then copied back to ZFS. But SR and FR can do that. In such cases, SnapRAID and FlexRAID offer better guarantees of data integrity than does ZFS.
 
OK, the first thing we must establish here is that I am not JoeComp nor do I agree with all his statements or claims. Please leave what he thinks and what he says out of the discussion when we are discussing my questions or things I said.

I have never said anything along the lines of other RAID systems being safer than ZFS, that's just silly.

My point is, if researchers say something, I dont really feel that I or you, must be able to explain their research, in order to quote researchers?

Not to quote them no you don't need to be able to exlpain it all

But to be able to draw your own conclusion about something that isn't in their paper (such as why my solution might not be as safe as ZFS) you need to be able to explain their research to be able to explain why an alternative particular solution doesn't work.

I know every big player has big problems with data corruption, so is the SnapRAID developer totally wrong when he claims it is safe? Why dont everyone buy his company and migrate to Snapraid? Does it make sense he is better than all researchers and engineers?

Here is my take on this part. Minus the SnapRAID stuff as I'm referring to the solution that I use to keep my data safe without the help of ZFS.

My thinking behind this is as follows.

The ZFS developers, the Amazon storage guys, the NetApp storage guys. They are looking for a storage solution that is usable for them and their business. This obviously means automatic, and a storage solution that makes good trades for how much space you are able to use compared to how much raw space you need to implement the system.

In my custom solution that I have described a bit before, my healing of the data is NOT automatic. It requires manual human intervention which can take a lot of time. Your question as to why the "big guys" wouldn't do it this way? My answer is because it is infeasible for a company like Amazon to have people hired to do manual data healing. There is too much data for humans to manage. So while my alternative may be just as safe, it cannot be used or adopted by them because of the sheer scale of what they need out of a system.

Secondly, my solution requires far more storage space. In fact, my solution doesn't even let you use half of your total storage space for usable space. I have 12TB of data, and my solution currently uses >16TB of "extra" data. That extra data consists of parity data, checksums, and full backup deduplicated data with infinite versions that are kept forever.

I think it's obvious why Amazon or other companies cannot adopt my storage solution. But just becasue they don't adopt it does not automatically mean my solution is not as safe. They don't adopt it because they can't afford to require more than twice the raw space as the files they are storing and they cant afford to support manual intervention and much poorer performance that happens when files must be manually restored.

I'm not trying to make claims. I am trying to ask questions and find out information.
I'm not saying my solution is better than ZFS. I am asking why it might not be. Because then I can improve it!

Your answer seemed to be: "because if your solution worked, then Amazon, CERN, etc would use it. So since they don't use it, then it must not work as well." But what if the reason they don't use it is not because it doesn't work, but because it makes huge tradeoffs that are infeasible for a larger scale operation to make. But maybe these tradeoffs are perfectally reasonable for a single home user (me) to make. So maybe my solution can be just as safe as ZFS, while being much worse in cases like performance and storage space efficiency. But then also being much more flexible in things like catestrophic recovery, data expansion, and cost. Which are the things most important to me, after resiliency of course.

If there are 2 solutions and both are just as safe, but one is cheaper and offers the flexability that is important to me I am going to choose that solution. Also so long as the negatives of that solution aren't a problem for me.

I don't believe there is a one-size-fits-all data storage solution. Data storage is all about managing risk, and I am able and willing to make compromises to things like speed and storage efficiency to gain things like cheaper cost and better expandability. Compromises that others (Amazon, etc) are not capable of making so they must turn to other solutions like ZFS that don't make those compromises, but do make other compromises that they can and are willing to make.
 
Last edited:
You are making assumptions and claiming I said something that I did not say.

I never said "SnapRAID is safe".
Here you say SnapRAID and FlexRAID is as safe as ZFS. ZFS has research on it's data integrity. If SnapRAID is equally safe, where is the evidence?
http://hardforum.com/showpost.php?p=1039900942&postcount=13
"Besides, SnapRAID and FlexRAID can detect and fix corruption as well as ZFS (just not in realtime)."


Note that this problem applies equally to ZFS -- if the data is generated, for example, on a system with NTFS (perhaps ripping a disc on a Windows machine) and then the data is copied to a ZFS fileserver, then ZFS cannot guarantee the integrity of the data (it could, for example, have been corrupted in the copy operation from NTFS to ZFS).
This is an example where I dont follow your logic. You are saying that if ZFS gets corrupted data from NTFS, then ZFS can not restore the data to it's uncorrupted state and therefore ZFS is not safe. This is not correct, ZFS guarantees that the data it gets is unaltered and intact as long ZFS has it. And that promise is kept. If you feed garbage into ZFS, ZFS promises the garbage will be intact and no bits are flipped on the garbage - and that promise is indeed kept. ZFS keeps it's promises to preserve the integrity of the data it has no matter how the data looks (even if it is garbage) - that is data integrity.

It is impossible for any storage solution that is fed corrupted data to restore it to the uncorrupted state. How could it do that, it would need to be magic! If I put a painting into a vault, then the vault promises that as long the painting is in the vault, no harm will come to the painting - and that promise is kept. The painting is unaltered so the vault is safe. You are asking for a magic vault like this: if I put a corrupted painting with lots of scars and trash - the vault must be able to restore the painting to it's original uncorrupted state or the vault is not safe! That would be impossible to ask of a vault. In the same vein, you can not ask that ZFS repairs the corrupted data it gets from NTFS, it would be magic. Integrity is to protect against changes. It seems that you dont understand what data integrity is. This misunderstanding from you, explains why I and you disagree so much on this matter.
 
Last edited:
In my custom solution that I have described a bit before, my healing of the data is NOT automatic. It requires manual human intervention which can take a lot of time. Your question as to why the "big guys" wouldn't do it this way? My answer is because it is infeasible for a company like Amazon to have people hired to do manual data healing. There is too much data for humans to manage. So while my alternative may be just as safe, it cannot be used or adopted by them because of the sheer scale of what they need out of a system.
Wouldn't it be safer if someone wrote a program that did your manual steps, and turned them into something fully automatic? A computer often does stuff faster and safer than a human. Humans are error prone.

I am convinced your manual steps are possible to program into a computer and do them automatically. If Oracle or NetApp or Amazon wanted to, they could replicate your behavior. What is it that you are doing, that a computer can not do automatically?
 
For large data, CERN, Amazon, etc - makes it very clear that just adding checksums does not make a storage solution safe. Just read their studies. Just because FlexRAID has some checksums does not mean it is safe.

Link to studies your referring to?

At some point any storage system relies on CPU/RAM, and you can find studies showing how these are not perfect, so therefore any storage system is also flawed. Is this the logic flow we're using to claim file checksums do not work? If so, how is ZFS immune to this?

Oh wait it is not:
5 In-memory data integrity in ZFS

In the last section we showed the robustness of ZFS to disk corruptions. Although ZFS was not specifically designed to tolerate memory corruptions, we still would like to know how ZFS reacts to memory corruptions, i.e., whether ZFS can detect and recover from a single bit flip in data and metadata blocks. Our fault injection experiments indicate that ZFS has no precautions for memory corruptions: bad data blocks are returned to the user or written to disk, file system operations fail, and many times the whole system crashes. This section is organized as follows. First, we briefly describe ZFS in-memory structures. Then, we discuss the test methodology and workloads we used to conduct the analysis. Finally, we present the experimental results and our observations.

http://research.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.pdf

So yeah ZFS is not safe either it turns out.
 
Link to studies your referring to?

At some point any storage system relies on CPU/RAM, and you can find studies showing how these are not perfect, so therefore any storage system is also flawed. Is this the logic flow we're using to claim file checksums do not work? If so, how is ZFS immune to this?

Oh wait it is not:
http://research.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.pdf

So yeah ZFS is not safe either it turns out.
I really dont agree with that viewpoint from the researchers. It is not the task of ZFS to protect against memory corruptions - that task is delegated to ECC ram. It is like the researchers are asking ZFS to protect against cpu corruptions - that is something outside ZFS domain and not ZFS responsibility. ZFS is a filesystem, not a system to guarantee RAM integrity, CPU integrity, integrity of TCP/IP traffic, etc. So, I dont agree with the researchers view point. Everyone knows that you should use ECC RAM with ZFS, to guard against RAM corruptions. RAM corruptions, CPU corruptions, TCP/IP corruptions are outside ZFS responsibility and it is wrong from the researchers to ask a filesystem to fix that too.

What do the paper say about ZFS abiility for data integrity on disks (not errors in the RAM sticks, CPU or TCP/IP)?

But, it is a good thing there are research on this, right? It feels a lot safer when ZFS has been scrutinized and examined by 3rd parts. There is another research paper on ZFS too, have you read it?
 
I don't know about you, but I'd prefer to have my data be uncorrupted, rather than have it guaranteed to be maintained at a certain level of corruption.

ZFS is not magic. ZFS is a tool. ZFS is an excellent tool for maintaining integrity of data that is generated in a system with ECC RAM and originally written to a ZFS filesystem, and then never copied outside of a ZFS filesystem.

However, ZFS is a poor tool for maintaining the integrity of data that is not originally written to a ZFS filesystem, or for data that is copied to a non-ZFS filesystem. SnapRAID or FlexRAID are superior to ZFS as tools for maintaining integrity of data that is generated on non-ZFS filesystems or data that is copied to non-ZFS filesystems, since they maintain checksum data independent of the underlying filesystem.

Here is a summary of some SnapRAID research:

The source code of SnapRAID has been examined. It computes checksums on any new data at a block level, with configurable block size, using Murmur3 or SpookyHash 128bit hashing algorithms whenever a "sync" command is run. When the SnapRAID "check" function is run, SnapRAID is able to determine either that the data is identical to that at the time the checksum was computed, or that the checksum of the current data does not match that computed at the time the data was added and a "sync" command run. Since the checksums of the data are contained entirely in multiple copies of a "content" data file, which has its own integrity verified with a CRC-32 checksum, it is possible to check the integrity of the data regardless of the filesystem the data is stored in, so long as at least one intact "content" file is available.
 
Last edited:
It is not the task of ZFS to protect against memory corruptions - that task is delegated to ECC ram.

Well even ECC is not fullproof. But I agree with you, in my opinion it is not ZFS's fault if the CPU/Memory are responsible for the data corruption. However ZFS does assume them to be faultless, and they are proven not to be.. at least in almost every implementation. That is why I ask you for link to study that says file checksums are not sufficient to detect corruption. As they too make the same assumption that the hardware is good. And I'd be interested in whether the study finds the fault in the actual checksum algorithm/implementation or the hardware it assumed to be good.

The former would be a real issue, the latter would be no different/worse than ZFS.
 
Back
Top