Data integrity: the risks of not using ECC with ZFS (a study)

Even AMD has been being a pain with ECC lately. The socket FM1 Llano CPUs would make excellent servers... if they supported ECC. I don't think they've included ECC support on anything newer than AM3+, and even there, plenty of boards don't include the extra traces between the CPU and memory sockets. Historicially, Gigabyte almost never includes them, Asus almost always does.

For some purposes, what I'd really love would be an atom-alike with ECC. But I think Intel knows that if they had an ECC capable Atom they'd lose a ton of money being unable to sell higher specced CPUs for some types of clustery apps. I'm somewhat amazed AMD hasn't tried to get into that market first. If neither of them move, ARM has a chance to take the massively redundant, ultra low power server market before either of them realize what has happened. But then again, ARM has been hinting at it for 2-3 years now, and as far as I know, nothing significant has happened...
 
The Cortex A-15 chip has ECC support.

What we need now is a port of ZFS to ARM. (FreeBSD?)

Cortex A-15 features:

* ECC support
* up to 1 TB RAM support (through 40-bit LPAE, already supported in linux)
* hypervisor for running multiple OSes simultaneously
* up to 4 MB L2 processor cache
* up to 2.5 GHz
* up to 8 cores
 
jwcalla said:
Is it true that if ZFS metadata gets corrupted, your entire pool is fubar'd? Because I wouldn't want that. That seems undesirable.
Click to expand...
It might be true. If any filesystem gets it metadata corrupted, then you are toast. But, ZFS stores the metadata on several places all over the raid/disk. It is not stored once. There are multiple copies everywhere. If one copy is corrupt, ZFS will look for the next copy. I think it is called... superblock(?).



cbutters said:
Everything I have read about ZFS points to that it absolutely loves RAM and will eat up as much RAM as it can:

http://wiki.freebsd.org/ZFSTuningGuide
First statement: "ZFS needs *lots* of memory."

http://constantin.glez.de/blog/2010...oracle-solaris-zfs-filesystem-performance#ram
"Add More RAM"

http://www.trainsignal.com/blog/zfs-nas-setup-guide
"ZFS system needs a minimum of 6 GB of RAM to achieve decent read/write performance and 8GB is preferred. For systems with more than 6TB of storage it is best to add 1GB of RAM for every extra 1TB of storage."

I don't know of any other file system that many people including _Gea recommend 6GB of RAM or more....
If I copy a bunch of data to my ZFS file server I can see the ram usage being totally used up. Its not a bad thing, it is doing its job, but if I make the same writes to a windows server box, I don't see anything close to the ram usage of ZFS.
Because of this I would say that ZFS is much more exposed to RAM bitflip errors on non-ecc memory since it is using more memory for the "same" operation (storing files)
Click to expand...
You are confusing things. It is a misconception that ZFS must have lot of RAM. ZFS prefers lot of RAM, but lot of RAM is not a must.

ZFS will automatically use all available RAM as disk cache. This is automatic and the disk cache can be turned off.

Other filesystems such as ext3 will not automatically start a disk cache. However, you can turn on disk cache for ext3 too, and then ext3 will use as much RAM as ZFS does. In Windows 7 all available RAM will be used as disk cache I think, so Windows 7 will use as much RAM as ZFS does.

To sum it up: ZFS will automatically use all RAM as a large disk cache to speed up things. ext3 will not automatically use all RAM as disk cache, but you can do that. In that case ext3 and ZFS will use as much RAM. If you turn off disk cache on ZFS, then ZFS will use as much ram as ext3. Here is more information:
http://en.wikipedia.org/wiki/ZFS#ZFS_cache:_L2ARC.2C_ZIL

I have used ZFS on 1GB RAM pentium 4 server for a year, without problems. Performance was low, but my data was safe.
 
tdg said:
However ECC isn't about going bad, it's about bit flips, which can and DO happen more than most realize. It won't always be the bit that was heading to your disks and thus affect your storage, but someday it will. Using ECC adds another layer of protection to storage, so when combined with ZFS it greatly increases data integrity. In fact if you go look at every Sun server and appliance they sell, they all come standard with ECC memory. When it comes to the utmost reliability of data, cheaping out and using non-ECC is just installing a weak link.
Click to expand...

good poast
 
brutalizer said:
It is a misconception that ZFS must have lot of RAM. ZFS prefers lot of RAM, but lot of RAM is not a must.
Click to expand...

I don't know about "must", but everyone who worked on ZFS talked about it having special RAM requirements. it's in the docs and the mailing lists and whatever else. ZFS was designed for servers with "lots" of RAM. that's not a misconception. and more RAM means more errors which means more need for ECC.
 
Most persons use ZFS because it offers a higher level of data security than other filesystems
I use ECC for the same reason regarding RAM. Both is needed to get at least informed about problems.

And if my OS shows Zero Byte free on heavy load, I know why have bought that much RAM
(To do work, not to idle). But ZFS has no problem running on a OS with 1 GB RAM.
 
Last edited:
brutalizer said:
.....



....
Other filesystems such as ext3 will not automatically start a disk cache. However, you can turn on disk cache for ext3 too, and then ext3 will use as much RAM as ZFS does
..........
I have used ZFS on 1GB RAM pentium 4 server for a year, without problems. Performance was low, but my data was safe.
Click to expand...

general usage of ext3 does not much read/write cache in the system, we can track down when ext3 created to solve major issues in ext2.
ext3 does not need a big cache in the system-> you can read http://lwn.net/Articles/283161/

you can use p4 with linux distro (No GUI) with ext3 for server purposes with NFS/SMB, it response very fast. the downside is P4 is power hunger and heat generator .
mdadm raid sits on the top of ext3.

we can not comparing ext3 and zfs since orange vs apple.
 
stewartjm said:
Even AMD has been being a pain with ECC lately. The socket FM1 Llano CPUs would make excellent servers... if they supported ECC. I don't think they've included ECC support on anything newer than AM3+, and even there, plenty of boards don't include the extra traces between the CPU and memory sockets. Historicially, Gigabyte almost never includes them, Asus almost always does.

For some purposes, what I'd really love would be an atom-alike with ECC. But I think Intel knows that if they had an ECC capable Atom they'd lose a ton of money being unable to sell higher specced CPUs for some types of clustery apps. I'm somewhat amazed AMD hasn't tried to get into that market first. If neither of them move, ARM has a chance to take the massively redundant, ultra low power server market before either of them realize what has happened. But then again, ARM has been hinting at it for 2-3 years now, and as far as I know, nothing significant has happened...
Click to expand...

What is the CPU in the HP Microserver? Isn't it Llano-based? It supports ECC.
 
iroc409 said:
What is the CPU in the HP Microserver? Isn't it Llano-based? It supports ECC.
Click to expand...

According to:
http://www.silentpcreview.com/HP_Proliant_MicroServer

It's a: 1.3gHz AMD Neo N36L

That architecture appears to be at least 2 years old, lots of articles in 2009, I don't remember it having much traction back then, a few netbooks and nettops. For light duty server use, it does appear like it would somewhat fill the Atom with ECC niche.

A Llano with ECC(if it existed) would be about as capable as an AM3+ dual or quad core, with a lower CPU+platform TDP. Not as nice as an 1156 or 1155 Xeon, but quite likely an excellent trade off for the CPU+MB price. I switched from AMD to Xeon for my newer servers for 2 reasons:
1. Intel was beating AMD on both performance and efficiency(especially idle efficiency).
2. There are a lot more, and better 1156 and 1155 Xeon motherboards with IPMI.

I'm pretty sure, that if Llano had ECC, and if the existence of LLano+ECC had led Tyan/SuperMicro to build some Llano board with IPMI, then I probably would've gone AMD instead of Intel on 1 or more of my server builds in the last 2 years.
 
cantalup said:
FUSE is not native :p , running on user space, native means running in kernel rings.

there is a porting to Native linux , where trying to work ong GPL. we should wait until really stable.
I prefer a stable release on real usage/server.
Click to expand...

Go to the first link in my post. Click on it and actually READ the page. See the note at the bottom that says "Native ZFS on Linux Produced at Lawrence Livermore National Laboratory"?

It is not the FUSE version. It is REAL, NATIVE ZFS for linux.
 
cantalup said:
general usage of ext3 does not much read/write cache in the system, we can track down when ext3 created to solve major issues in ext2.
ext3 does not need a big cache in the system-> you can read http://lwn.net/Articles/283161/

you can use p4 with linux distro (No GUI) with ext3 for server purposes with NFS/SMB, it response very fast. the downside is P4 is power hunger and heat generator .
mdadm raid sits on the top of ext3.

we can not comparing ext3 and zfs since orange vs apple.
Click to expand...

bAMtan2 said:
I don't know about "must", but everyone who worked on ZFS talked about it having special RAM requirements. it's in the docs and the mailing lists and whatever else. ZFS was designed for servers with "lots" of RAM. that's not a misconception. and more RAM means more errors which means more need for ECC.
Click to expand...



Let me try to explain again. The reason ZFS uses lot of RAM, is because ZFS uses a disk cache, called ARC. If you switch off the disk cache, then ZFS will not use more RAM than any other filesystem. ZFS can run without a disk cache, just as ext3 does.

If the OS needs more RAM, then ZFS disk cache will release RAM. ZFS will use all free RAM as disk cache, but will immediately release the RAM if some application asks for it.

It is common usage to use disk cache to increase performance. But that is your choice if you want to do. You can also install and use a disk cache for ext3 if you wish, but no one would say that ext3 _must_ use lots of RAM. A large disk cache increases performance, but it is not needed, it is not a must.

Was this explanation clearer?
 
brutalizer said:
Let me try to explain again. The reason ZFS uses lot of RAM, is because ZFS uses a disk cache, called ARC.
Click to expand...

yes, but zfs was designed and tested on systems with large quantities of memory, and the ARC enabled, and it is unusually slow and bad and potentially buggy without that "lots" of RAM. so the memory demands are not a misconception, they are the status quo. if you are having success with low memory then that is great.
 
Not true, I get 500MB/s read speed with only 2GB ram. ZFS do not require a shit load of memory.
 
I think most of the memory "ohhh noes!" were related to the de-dupe feature.
 
Yeah supposedly the de-dupe only starts to have decent performance at 32-64 gigs or more. Basically if it can't cache most/all of the de-dupe block hashes in memory then performance drops like a rock.

But it sounds like de-dupe in general is still pretty flaky, so I've not even though about trying it.
 
bAMtan2 said:
yes, but zfs was designed and tested on systems with large quantities of memory, and the ARC enabled, and it is unusually slow and bad and potentially buggy without that "lots" of RAM. so the memory demands are not a misconception, they are the status quo. if you are having success with low memory then that is great.
Click to expand...
It is true that ZFS is for enterprise storage servers with many disks and lot of RAM, yes. But that does not mean ZFS will not work on low ram PCs. ZFS works well on low RAM PCs. Just turn off the ARC disk cache and ZFS will work as good as ext3 or any other filesystems.

Why do think ZFS is "unusually slow and bad and buggy without lots of RAM"? Do you have any links that supports this claim, or is it wishful thinking from your side?

Let me try to explain again, ZFS works best with large disk cache because then performance sky rockets. But ZFS works fine without disk cache. Many people use ZFS on low ram PCs without problems. No bugs here.
 
I think the ZFS Evil Tuning Guide includes configuration recommendations for systems with less than 2GB of RAM.
 
brutalizer said:
Let me try to explain again. The reason ZFS uses lot of RAM, is because ZFS uses a disk cache, called ARC. If you switch off the disk cache, then ZFS will not use more RAM than any other filesystem. ZFS can run without a disk cache, just as ext3 does.

If the OS needs more RAM, then ZFS disk cache will release RAM. ZFS will use all free RAM as disk cache, but will immediately release the RAM if some application asks for it.

It is common usage to use disk cache to increase performance. But that is your choice if you want to do. You can also install and use a disk cache for ext3 if you wish, but no one would say that ext3 _must_ use lots of RAM. A large disk cache increases performance, but it is not needed, it is not a must.

Was this explanation clearer?
Click to expand...

you can not compare ext3 with zfs, they are not the same animal :p.

in real environment, most people just use default ext3 parameters, it just works., with just some minor tweaking.

please refer to me to any articles or forums that ext3 performance is increased when use a bigs disk cache(memory), could you?

zfs will utilized memory (cache and on the fly calcuation in memory), Not on ext3 (legacy linux filesystem)

on OS side, memory are handled in the kernel ring 0 or 1 ( need to look the book), OS can manage memory for example, release memory, claim as cache/buffer memory, and others). this is Not related with filesytem (zfs, ext3, or even hammerfs),

my restatement is:
we are talking on ZFS will give us another security from ram flip-flop, If we are using ECC ram, not less and not more
 
Last edited:
kristof said:
Go to the first link in my post. Click on it and actually READ the page. See the note at the bottom that says "Native ZFS on Linux Produced at Lawrence Livermore National Laboratory"?

It is not the FUSE version. It is REAL, NATIVE ZFS for linux.
Click to expand...

yes, native but not in any real environment as I know. I mention in my previous posting

I already explain native zfs is on progress since solaris zfs license is not compatible with linux gpl. luckyly some people are porting/forking to linux

please read my posting again, you do not need using CAPSLOCK, I already follow native zfs linux since 2010 :D.

you can read on http://zfsonlinux.org/ , from you posting.
you can read porting zfs to linux disclaimer http://zfsonlinux.org/zfs-disclaimer.html

as I know today, none uses native zfs linux in a real working environment :).

you could check on http://www.osnews.com/story/23416/Native_ZFS_Port_for_Linux
Main developer Brian Behlendorf has also stated that the Lawrence Livermore National Laboratory has repeatedly urged Oracle to do something about the licensing situation so that ZFS can become a part of the kernel. "We have been working on this for some time now and have been strongly urging Sun/Oracle to make a change to the licensing," he explains, "I'm sorry to say we have not yet had any luck."

on the other side, you can start to look btrfs (oracle open source, which is in limbo due on oracle :p)
 
cantalup said:
as I know today, none uses native zfs linux in a real working environment :).
Click to expand...

What does that have to do with the question of whether or not native zfs for linux exists? You can't say that something doesn't exist because you don't know an example of it "in a real working environment." The code is there. It exists. People are running it right now.
 
kristof said:
What does that have to do with the question of whether or not native zfs for linux exists? You can't say that something doesn't exist because you don't know an example of it "in a real working environment." The code is there. It exists. People are running it right now.
Click to expand...

I said "as I know". ymmv
people or working real servers?
you can reread my posting, natvie zfs is on working... until native linux zfs can achieve stability as solaris zfs that is running on working real servers
real working environment, for your server that serving your business 24/7 ideally.
you do not want to update or patch frequently on your working server due on linux zfs bug fixes. the occasional update or patch is reasonable since nothing is perfect
I prefer running solaris zfs than linux zfs for business matters

you ca pick native zfs linux, if prefer, there is nothing to stop you for using open source..unless your boss :D
 
Last edited:
cantalup said:
you can not compare ext3 with zfs, they are not the same animal :p.

in real environment, most people just use default ext3 parameters, it just works., with just some minor tweaking.
Click to expand...
The same with ZFS. No tweaking here. It is recommended to not tweak ZFS. There is a guide called "evil tweaking ZFS guide" - which says that no one should tweak ZFS. But if you really need to, consult the guide (which is evil to do, because tweaking is evil).


please refer to me to any articles or forums that ext3 performance is increased when use a bigs disk cache(memory), could you?
Click to expand...
I dont have such articles, but it is common sense. A disk cache in RAM will always be faster than if you are not using disk cache at all. Agreed?

A cpu with large cpu cache 16MB, will always be faster than a cpu without any cpu cache. The same with a hard disk, if it has 64MB cache or a no cache, then cache will always be faster. It is common sense that RAM is always faster than slow hard disks. Agreed?


zfs will utilized memory (cache and on the fly calcuation in memory), Not on ext3 (legacy linux filesystem)
Click to expand...
ext3 can also use disk cache. I believe that ext3 is automatically using disk cache, it should. Even Windows7 is using available RAM as disk cache. I can not imagine Linux systems does not use RAM disk cache? Every modern system should use disk cache. Maybe ext3 has limited disk cache up to 1GB RAM or so. But every modern OS should use disk cache to increase performance. Caches are vital and extremely important to increase performance. You can not get high performance without cache. Every part in the computer use cache; the cpu, the disk drive; etc


my restatement is:
we are talking on ZFS will give us another security from ram flip-flop, If we are using ECC ram, not less and not more
Click to expand...
My statement is; ZFS use slightly more RAM than other filesystems. But not huge amounts of RAM.

You can probably use ext3 on a 256MB Linux PC. You can use ZFS on 512 MB PC. You must not use many GB of RAM. If you consider 512MB RAM as "huge amounts" then fine. But ZFS does not need many GB. You should try ZFS on a 512 MB pc and see yourself.
 
cantalup said:
I said "as I know". ymmv
people or working real servers?
you can reread my posting, natvie zfs is on working... until native linux zfs can achieve stability as solaris zfs that is running on working real servers
real working environment, for your server that serving your business 24/7 ideally.
you do not want to update or patch frequently on your working server due on linux zfs bug fixes. the occasional update or patch is reasonable since nothing is perfect
I prefer running solaris zfs than linux zfs for business matters

you ca pick native zfs linux, if prefer, there is nothing to stop you for using open source..unless your boss :D
Click to expand...

This is not a professional san builders forum. It's is hobbyist forum. People here build zfs servers for their own enjoyment. Most people here do not mind using hardware and software that wouldn't be acceptable in the enterprise.

We do have the occasional zfs question where someone is building something "for work" but the overwhelming majority of zfs builds are home servers for tv, movies, etc.
 
The risks of using ZFS without ECC RAM will be the same as using any RAID, LVM or filesystem implementation without ECC. My attitude is this: if, for a given application, you would use a hardware RAID controller over software RAID, and you would use ZFS over a hardware RAID controller, you should use ECC RAM.

Hardware RAID comes with ECC RAM for a reason. :)
 
kristof said:
This is not a professional san builders forum. It's is hobbyist forum. People here build zfs servers for their own enjoyment. Most people here do not mind using hardware and software that wouldn't be acceptable in the enterprise.

We do have the occasional zfs question where someone is building something "for work" but the overwhelming majority of zfs builds are home servers for tv, movies, etc.
Click to expand...

as I said on my previous posting.
you can pick ..... open source is beautiful :D, no one can dictate you..(unless ur boss ..:p)

I just give my knowledge and understanding, where everyone can at leas have a plan to go further in years later without giving a lot of headaches. those my comments are arguable with common senses + technical knowledge.


some of us are using for soho or small dataserver, not all for tv or movies...
 
parityboy said:
The risks of using ZFS without ECC RAM will be the same as using any RAID, LVM or filesystem implementation without ECC. My attitude is this: if, for a given application, you would use a hardware RAID controller over software RAID, and you would use ZFS over a hardware RAID controller, you should use ECC RAM.

Hardware RAID comes with ECC RAM for a reason. :)
Click to expand...

:D, I already explained on my first or two previous post(s)

I try to steer away that ECC is useless....

as I said on my previous posting, better using ECC for any software RAID implementation for avoiding headaches in the future :D
 
Last edited:
brutalizer said:
I dont have such articles, but it is common sense. A disk cache in RAM will always be faster than if you are not using disk cache at all. Agreed?

A cpu with large cpu cache 16MB, will always be faster than a cpu without any cpu cache. The same with a hard disk, if it has 64MB cache or a no cache, then cache will always be faster. It is common sense that RAM is always faster than slow hard disks. Agreed?



ext3 can also use disk cache. I believe that ext3 is automatically using disk cache, it should. Even Windows7 is using available RAM as disk cache. I can not imagine Linux systems does not use RAM disk cache? Every modern system should use disk cache. Maybe ext3 has limited disk cache up to 1GB RAM or so. But every modern OS should use disk cache to increase performance. Caches are vital and extremely important to increase performance. You can not get high performance without cache. Every part in the computer use cache; the cpu, the disk drive; etc



My statement is; ZFS use slightly more RAM than other filesystems. But not huge amounts of RAM.

You can probably use ext3 on a 256MB Linux PC. You can use ZFS on 512 MB PC. You must not use many GB of RAM. If you consider 512MB RAM as "huge amounts" then fine. But ZFS does not need many GB. You should try ZFS on a 512 MB pc and see yourself.
Click to expand...

as I said ZFS and ext3 is different apple
1) ext3 does not built-in software-raid, mdadm is the guy for doing software radi
2) ZFS has built-in software-raid

if you compare 512M on linux native zfs and ext3, I believe... ext3 will get better performance, since nothing additional layer.. just a simple filesystem
try yourself...

ext3 does not use diskcache,
the OS provide thecache, and it is not big cache.
ext3 does rely on barrier and journaling(No checksum check)

modern OS not dictated for using disk cache or not. the implementation dictate what to use and what to do as long as kernel ring (1 and) can provides the request.

for example: ext3 do not request write cache. OS just give a small buffer which depends on paramateres during ext3 created as a default.
on the other side is, OS implementation is different among them, Linux is mostly adhered to POSIX, solaris has different approach

Linux does use RAM read/write cache (not disk cache) and not big. you can follows on native zfs linux implementation where they have some limitations and ways for workarounds :D
 
Any idea what's going on...all of a sudden ZFS server gets in mode where all write activity is severely blocked...see test below..takes 3 seconds to write 32K. Nothing else really running on the box...

Code: 
[Sun May 06 05:40:39 root@zulu01:~/DTraceToolkit-0.99 ] # lockstat -kIW -D 20 time dd if=/dev/zero of=/rz2pool/cytel/temporary/ddtest7.dd bs=32000 count=1
1+0 records in
1+0 records out

real        2.9
user        0.0
sys         0.0

Profiling interrupt: 2312 events in 2.987 seconds (774 events/sec)

Count indv cuml rcnt     nsec Hottest CPU+PIL        Caller                  
-------------------------------------------------------------------------------
 2141  93%  93% 0.00     4656 cpu[6]                 cpu_idle                
  140   6%  99% 0.00     2940 cpu[4]                 avl_destroy_nodes       
    8   0%  99% 0.00     2508 cpu[4]                 mutex_enter             
    4   0%  99% 0.00     2793 cpu[4]                 kmem_cache_free         
    4   0%  99% 0.00     2459 cpu[4]                 mutex_exit              
    2   0%  99% 0.00     2438 cpu[4]                 kmem_free               
    1   0%  99% 0.00     5630 cpu[0]                 fsflush_do_pages        
    1   0% 100% 0.00    13029 cpu[2]                 avl_rotation            
    1   0% 100% 0.00    23998 cpu[1]+11              _resume_from_idle       
    1   0% 100% 0.00     5856 cpu[0]                 page_nextn              
    1   0% 100% 0.00     2475 cpu[0]                 atomic_cas_64           
    1   0% 100% 0.00     5649 cpu[1]                 prefetch_page_r         
    1   0% 100% 0.00     4312 cpu[0]                 tsc_read                
    1   0% 100% 0.00      528 cpu[7]+5               ddi_get32               
    1   0% 100% 0.00    96645 cpu[7]+5               sd_return_command       
    1   0% 100% 0.00     3017 cpu[4]                 space_map_vacate        
    1   0% 100% 0.00    21462 cpu[5]                 arc_write_done          
    1   0% 100% 0.00     5658 cpu[2]                 (usermode)              
    1   0% 100% 0.00    17515 cpu[3]                 scsi_init_pkt           
-------------------------------------------------------------------------------

Code: 
# .hotkernel

genunix`avl_remove                                         12   0.1%
unix`mutex_enter                                           21   0.2%
genunix`avl_rotation                                       38   0.4%
zfs`space_map_remove                                      110   1.1%
genunix`avl_insert                                        206   2.0%
zfs`space_map_seg_compare                                 232   2.3%
genunix`avl_walk                                          241   2.3%
zfs`metaslab_segsize_compare                              270   2.6%
genunix`avl_find                                          346   3.4%
unix`cpu_idle                                            8683  84.4%
 
cantalup said:
as I said ZFS and ext3 is different apple
1) ext3 does not built-in software-raid, mdadm is the guy for doing software radi
2) ZFS has built-in software-raid

if you compare 512M on linux native zfs and ext3, I believe... ext3 will get better performance, since nothing additional layer.. just a simple filesystem
try yourself...

ext3 does not use diskcache,
the OS provide thecache, and it is not big cache.
ext3 does rely on barrier and journaling(No checksum check)

modern OS not dictated for using disk cache or not. the implementation dictate what to use and what to do as long as kernel ring (1 and) can provides the request.

for example: ext3 do not request write cache. OS just give a small buffer which depends on paramateres during ext3 created as a default.
on the other side is, OS implementation is different among them, Linux is mostly adhered to POSIX, solaris has different approach

Linux does use RAM read/write cache (not disk cache) and not big. you can follows on native zfs linux implementation where they have some limitations and ways for workarounds :D
Click to expand...
I do believe that ext3 has higher performance than ZFS, if no one use disk cache. The reason is that ZFS does lot of checksumming to protect your data which eats up cpu. ext3 does not have checksums, so not much cpu burning and no protection of data.

However, ext3 scales bad (because ext4 scales bad, see link below). ZFS scales extremely well. It is often like this;
-Naive implementation is fast on small working set, single disk. Does not scale well up to many disks. Not much RAM nor cpu is used.
-Complex implementation is slow on small working set. Scale well to many many disks. More RAM and cpu is used.

For instance, there is a 55 petabyte ZFS installation that the new IBM supercomputer will have. The supercomputer will use Lustre for raiding, but it will use ZFS as filesystem. Earlier, Lustre used ext4 as filesystem but ext4 scaled bad. Here are benchmarks, ext4 vs ZFS on very large Petabyte installations. ZFS shines and outperforms hardware raid. ext3 is very slow on large installations.
http://hardforum.com/showthread.php?t=1691239



So the point is: you need not many GB of RAM to use ZFS. You can switch off disk cache in ZFS, and then a PC with 512MB RAM is fine. If people says: "you must have many GB of RAM to use ZFS" - that is a false statement and not true.
 
parityboy said:
The risks of using ZFS without ECC RAM will be the same as using any RAID, LVM or filesystem implementation without ECC.
Click to expand...


That's not strictly true.......if you get a bitflip in non-ECC memory, ZFS might detect it - "standard" filesystems won't!

That's not to say ECC isn't worthwhile, but it also depends on how you use your system, and what you use it for.

If you accept that ECC memory is needed on a server, then you must also accept that ECC memory is needed on any machine which stores data on or uses data from said server.
 
Have a look at

http://hydra.geht.net/tino/howto/zfs/blog/lifesaver/mem/


The bottom line is that it will depend on when the bitflip happens.
On a write, if it happens after the checksum is calculated, then it would be detected on read as the checksum won't match. If it happens before the checksum is calculated, then it probably wouldn't be detected.
However, you can protect against this by doing verified copies to the server - there are various methods of doing this - arguably something we should all be doing anyway!

On a read, if a bitflip happens within some data after that data has been checksummed and is in memory, then ZFS won't detect this - this is where ECC memory comes in.
That said, it also depends on what the data is, as to how much effect such a bitflip would have.
On a home media server for instance, the effect might be negligible anyway.
 
Ok, that was a very good link. He uses ZFS on Linux and has much interesting info there.
 
So what do I use for a 5TB personal media server and some important files?

ZFS raidz1 or something else?

Presenting the storage to ESXi
 
DlStreamnet said:
Raidz 2 doesn't protect against bitflip errors though does it? Neither does raidz1.

This storage OS question is driving me mad for non-ecc systems.
Click to expand...

There are thousands of reasons to loose data.
ZFS can avoid 70% of them. ECC adds 10%

(Percent values are only estimated)
 
DlStreamnet said:
Raidz 2 doesn't protect against bitflip errors though does it? Neither does raidz1.

This storage OS question is driving me mad for non-ecc systems.
Click to expand...


ZFS "might" detect the effects of a bitflip though!

At the end of the day, no filesystem or RAID scheme can really protect against this - if it's a big worry then you'll need ECC memory (correctly implemented).

If it's a media server, you can protect against bitflips on data upload by using verifed copies - this isn't practical though on data which changes frequently (eg live active database).
 
You must log in or register to reply here.
Back
Top