Looking for info on ZFS

W

white_devil

Limp Gawd
Joined
Apr 21, 2007
Messages
229
I've been reading through the threads on here and have seen several people talk about ZFS and offer to go into more detail if someone asked, so I'm asking :). I'm currently running a hardware RAID 5 and am about to set up a hardware RAID 6 but maybe ZFS is the way to go instead. If someone would post some info a layperson could understand that would be great. I've set up quite a few RAIDs using 3ware and LSI controllers (and various onboard) and understand RAID fairly well but if there's something better I'm game. Is this something a Linux novice could set up without too much trouble?
 
First you need solaris, opensolaris, freebsd or freenas. The linux zfs-fuse implementation is too slow to use.

Then you need to get rid of the hardware controllers or put them in JBOD mode. Motherboards ports are preferred. If you are doing all of this hardware raid you probably are using windows. I suggest you try freenas as its designed to be not that difficult to manage. I suggest you install virtualbox and testdrive it under virtual box. Just create a few virtual drives in VB and test the system.
 
You could do OpenSolaris/FreeBSD for sure; but if you want something with a nice web-GUI like FreeNAS, then that would mean an older (and still experimental) version of ZFS offered at this time. It will take a while still before FreeNAS with ZFS version 13/14 get's released; until that time the ZFS feature offered by FreeNAS is still experimental; without all the bug fixes and stability fixes that originate from OpenSolaris that made it into FreeBSD 8.0 and 8.1. FreeNAS is still based on FreeBSD 7.x.

The idea of ZFS is relaxed hardware requirements; you don't need a hardware RAID controller; just full-bandwidth SATA ports. You can use a HW RAID controller as 'normal' SATA controller; that's how i use my Areca now simply as 'dumb HBA' in passthrough mode. Funny thing is; the passthrough devices still behave as arrays: if there is a cabling error or bad sector, Areca would 'destroy' that disk and generally screw with my OS by removing the disk/array and returning only errors. When i reboot, the passthrough array was gone; i had to re-create it again and then ZFS would instantly recognise it again. What this tells me? That you would preferably want a 'dumb HBA' controller; such as the SuperMicro USAS-L8i controller; which provides 8x SATA/300 at full bandwidth, with being able to disable all RAID mode and act as normal non-RAID SATA controller in AHCI mode.
 
Im considering ZFS too, mainly to learn. I noticed relaxed hardware requirements were mentioned. How relaxed are we talking? I have a really old box with a 550 P3 in it ( Im assuming thats too old) and another I could spare for play purposes thats got a P4 3.06 in it.

Also, it seems to get the "full" experience it would be best to run Solaris/OpenSolaris. Is this correct ? And also, isnt Solaris pretty integrated with the hardware? I thought I read somewhere in my internet ramblings that it works best on certain hardware so I am confused. Does it work at all on unspecified hardware or is that a roll of the dice? Thanks in advance, I read about ZFS a long long time ago but at the time didnt have the money or time to even bother with it, kudos to you guys that are letting the people know about it.
 
ZFS is brilliant

I throw on opensolaris b134 (from genunix.org) onto plain jane intel core 2 duo E8400 + asus intel 3200 mobo (shit) + 2x4gb ram (osol loves ram) and as many hard drives as you can fit + whatever raid hw cards for more ports

ZFS gives you
- deduplication
- compression
- snapshots
- ability to use cheap disks
- only 2 commands to learn, zpool and zfs
Anyone still using hardware raid needs to be welcomed to the 21st century
anyone using other software raid is suicidal.
 
anyone using other software raid is suicidal.
Click to expand...

I agree with everything but this one. linux software raid is extremely robust. I would say its more robust than most hardware raid.
 
I do not find linux raid complex at all.

I know zfs is good but only if it would not require me changing my 20+ servers from linux to
freebsd. And zfs-fuse is not an option. I tried that. Way way too slow. As for btrfs, I do not expect that to be stable any time soon well unless someone puts it in their enterprise linux and puts a better effort in the development and testing.
 
drescherjm said:
I do not find linux raid complex at all.

I know zfs is good but only if it would not require me changing my 20+ servers from linux to
freebsd. And zfs-fuse is not an option. I tried that. Way way too slow. As for btrfs, I do not expect that to be stable any time soon well unless someone puts it in their enterprise linux and puts a better effort in the development and testing.
Click to expand...

Different situations, different solutions

Side note: I prefer osol to fbsd because of support, fbsd zfs is still old version, zfs v2x isn't scheduled for freebsd 9 or later! :(

Avoid zfs-fuse on linux, it's more hassle than it's worth
 
vraa said:
But overly complex and limited in it's use

I believe the btreefs might be handy for linux users, but still too early in development phase for production use

Edit: I highly recommend over viewing this link before using zfs/osol
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
Click to expand...

Linux Software RAID is trivial to use and does Software RAID extremely well. It supports RAID 0, 1, 5, 6, and 10, among others. Given what it is, what's not to like?
 
drescherjm said:
As for btrfs, I do not expect that to be stable any time soon well unless someone puts it in their enterprise linux and puts a better effort in the development and testing.
Click to expand...

I give BTRFS a year before its an option on enterprise Linux and 2 years before it is the default install. IIRC, it'll be an optional FS in a default Fedora 13 install. All the major Linux distros want it. All Linux users want it. It'll happen faster than you think. :)
 
lizardking009 said:
Linux Software RAID is trivial to use and does Software RAID extremely well. It supports RAID 0, 1, 5, 6, and 10, among others. Given what it is, what's not to like?
Click to expand...

As someone who ran linux software raid for 3 years before being burned by it and going to hardware raid under windows, let me give you some reasons:

1) lvm2 is a real pain - it cuts performance and makes recovery of certain kinds of faults very hard.

2) device nomenclature in linux software raid is pretty difficult if you have a lot of disks in a hot swap environment. Device names are changing when you swap devices this can easily lead to a situation where you pull the wrong disk - linux software raid does not allow you to light activity LED's etc... so you know which disk is the one you are supposed to work on.

3) Controller or cabling errors can cause multiple disks to go offline at once, and getting the array back up in that case is not automatic. And the processing for resyncing through the use of the -force option during assembly is prone to problems that can cause you to lose the array or large chunks of it.

If you have a fixed config without not too many disks and everything works with an occasional disk going completely bad as the failure mode, then linux raid works fine. But if you have more than 10 or 12 disks and have to deal with disks slowly failing things can be a real pain to manage.

The core linux tools are quite reliable, but the overall manageability sucks big time. The hardware raid guys have tools that are light years better, and while zfs doesn't have a good GUI for management, it handles various problem scenarios way better than md does.

Trust me, I am a unix hack from years back and are extremely comfortable with the shell and the kernel. But life is pretty hard on you when the complexity gets high linux software raid. Odd problems do not get handled so well.
 
1) lvm2 is a real pain - it cuts performance and makes recovery of certain kinds of faults very hard.
Click to expand...

I do not see much performance loss from lvm. To me its easy to use. I use it on dozens of servers. I never use it to span non raided volumes though. Also LVM is not required.

2) device nomenclature in linux software raid is pretty difficult if you have a lot of disks in a hot swap environment. Device names are changing when you swap devices this can easily lead to a situation where you pull the wrong disk - linux software raid does not allow you to light activity LED's etc... so you know which disk is the one you are supposed to work on.
Click to expand...

I agree it is sometimes difficult to figure out what drive needs to be pulled. I do this by serial number if I am not using a hot swap bay with leds. On systems with leds on the hot swap bays I use hdparm to cause activity on the disks to help find the drive I need.

But if you have more than 10 or 12 disks and have to deal with disks slowly failing things can be a real pain to manage.
Click to expand...

I have many systems with 6 to 8 drives with 1 system with 20 sata drives.

but the overall manageability sucks big time. The hardware raid guys have tools that are light years better
Click to expand...

I disagree on both counts. All of my servers are monitored by nagios. I get instant alerts if anything goes wrong with the raid also I get alerts for smart changes like reallocated sectors and other system problems. This is at least as good as my arcea controllers in windows. mdadm is not hard to use.
 
Last edited:
Have you tried running lvm2 to span raid volumes? It's at least a 50% hit on streaming performance. Pretty much everyone on the mailing lists will verify this too - but it's the ability to recover from underlying volume errors that can take down the logical volume that's teh real killer.

I'm not sure how doing swaps based on serial number helps. Any kind of hot swap tray mounting will prevent you from being able to inspect the disk and be able to read the serial number from it. The bets you can do is write the serial number on the outside part of the tray and make sure you always keep the labels and trays up to date. This is a really clunky approach compared to being able to blink an LED to indicate a disk failure, or even having consistent device nomenclature.

By management I am not talking about being able to read smart errors, but being able to deal with arrays that have members that are throwing errors. My experience is that md will dump a device that throws an error, forcing you to resync it manually, as opposed to failing that sector and automatically repairing it the way ZFS and the hardware raid controllers work.

The lack of a decent GUI that gets into this level of detail is sorely missed. No linux GUI that I am aware of supports you doing reassembly operations, the -force command, or even recording the device order of an array that was created and modified so you can manually reassemble it if needed. The hardware vendors GUI tools do all this and more.
 
Have you tried running lvm2 to span raid volumes? It's at least a 50% hit on streaming performance.
Click to expand...

Yes. On the 3 year old athlon X2 system which I have lvm spanning raid arrays I get 300+MB/s reads and 200+MB/s writes so its definitely not a 50% hit.
 
drescherjm said:
All of my servers are monitored by nagios. I get instant alerts if anything goes wrong with the raid also I get alerts for smart changes like reallocated sectors and other system problems.
Click to expand...

What nagios plug-ins or scripts do you use to do all of that? I have seen a nagios plug-in that runs an mdadm volume check, but I did not see one that checks for "anythng going wrong" or that does smartctl queries and tracking.
 
drescherjm said:
Yes. On the 3 year old athlon X2 system which I have lvm spanning raid arrays I get 300+MB/s reads and 200+MB/s writes so its definitely not a 50% hit.
Click to expand...

Software raid can run VERY fast. Just because you are getting good rates doesn't mean lvm2 is not slowing things down a lot.
 
Thanks for all the info, even the thread derail since I learned some stuff from that too. So if I want to try my luck with an OpenSolaris or FreeBSD implementation how hard is it going to be to set up for someone with very little *nix experience? Is this something that can be done through a GUI or will I be stuck doing it through CLI? I actually prefer learning how to do things through CLI but time is at a premium these days and I don't see myself finding the time to learn considering I'd be starting from scratch. Also, how hard is it to set up email alerts for various drive events?

I noticed relaxed hardware requirements were mentioned. How relaxed are we talking?
Click to expand...
I'm wondering this too. I have a lot of old boxes collecting dust that I'd like to do something with. Would an old P4 or Athlon XP be able to handle something like this or will I need something beefier?
 
white_devil said:
Thanks for all the info, even the thread derail since I learned some stuff from that too. So if I want to try my luck with an OpenSolaris or FreeBSD implementation how hard is it going to be to set up for someone with very little *nix experience? Is this something that can be done through a GUI or will I be stuck doing it through CLI? I actually prefer learning how to do things through CLI but time is at a premium these days and I don't see myself finding the time to learn considering I'd be starting from scratch. Also, how hard is it to set up email alerts for various drive events?


I'm wondering this too. I have a lot of old boxes collecting dust that I'd like to do something with. Would an old P4 or Athlon XP be able to handle something like this or will I need something beefier?
Click to expand...

I can't speak too much about freebsd implementation, but on opensolaris everything can easily be done through CLI so you can do it headless
You only need to know 2 commands
zfs and zpool

Re: email alerts, try this
http://prefetch.net/code/fmadmnotifier
Otherwise a good ol' cron script can do it
zpool status -x | grep -v 'all pools are healthy'

You can do 32bit osol, but zfs benefits from gobs of ram and multicore CPUs
Be an experimenter, get the oldest shittiest hardware you can find and see if stuff still works :D

Here are some links I strongly suggest you skim over so you can get a rough idea of how to use ZFS

http://lildude.co.uk/zfs-cheatsheet
http://hell.jedicoder.net/?p=45
http://uadmin.blogspot.com/2005/11/cool-zfs-tricks.html
http://www.unixpowered.com/blog/2008/12/20/quick-and-dirty-zfs-cheatsheet/

Do not be afraid to ask questions :)
 
I will copy paste from wikipedia
http://en.wikipedia.org/wiki/Non-standard_RAID_levels#RAID-Z

Sun's ZFS implements an integrated redundancy scheme similar to RAID 5 which is called RAID-Z. RAID-Z avoids the RAID 5 "write hole"[6] by its copy-on-write policy: rather than overwriting old data with new data, it writes new data to a new location and then atomically overwrites the pointer to the old data. It avoids the need for read-modify-write operations for small writes by only ever performing full-stripe writes; small blocks are mirrored instead of parity protected, which is possible because the file system is aware of the underlying storage structure and can allocate extra space if necessary. There is also RAID-Z2 which doubles the parity structure to achieve results similar to RAID 6: the ability to sustain up to two drive failures without losing data.[7] In July 2009, triple-parity RAID was added to OpenSolaris.[8][9]
Click to expand...

Super super cliffnotes = 1/xth of your drives is parity where x = number of drives you have
Raid2z = 2/xth of your drives...
 
Basically, many goodness of ZFS comes from the fact that ZFS is both filesystem and RAID-engine; that allows for things that were not possible with conventional (hardware) RAID. For example dynamic stripesizes are impossible with hardware RAID; but ZFS can use it because its both RAID engine and filesystem in one.

Emerging properties, i would say. :)
 
sub.mesa said:
The idea of ZFS is relaxed hardware requirements; you don't need a hardware RAID controller; just full-bandwidth SATA ports. You can use a HW RAID controller as 'normal' SATA controller; that's how i use my Areca now simply as 'dumb HBA' in passthrough mode. Funny thing is; the passthrough devices still behave as arrays: if there is a cabling error or bad sector, Areca would 'destroy' that disk and generally screw with my OS by removing the disk/array and returning only errors. When i reboot, the passthrough array was gone; i had to re-create it again and then ZFS would instantly recognise it again. What this tells me? That you would preferably want a 'dumb HBA' controller; such as the SuperMicro USAS-L8i controller; which provides 8x SATA/300 at full bandwidth, with being able to disable all RAID mode and act as normal non-RAID SATA controller in AHCI mode.
Click to expand...

Sub:
As you know I'm about to build a new system based on opensolaris. I have carefully checked all the hardware compatibility lists to ensure I don't run into problems. Today I expect the guy in the brown truck to deliver a Norco 4220, ARECA|ARC-1680X, Cables, and I already have the HP Expander and some drives. Should I not open the Areca and return it? I would need a low profile card with external ports (coming from the expander) to use in my Tyan server. What card would you suggest that is Opensolaris / FreeBDS compatible? If I can save @ $600 please let me know..
Thanks again..

Jeff
 
SuperMicro USAS-L8i is a 2x Mini-SAS controller that allows 8 SATA/300 devices to be connected as 'dumb' HBA; normal AHCI SATA controller without RAID.

That's most ideal to ZFS i believe. The Areca 1680iX is running with an internal port multiplier as far as i know; if you connect another port multiplier to that; well i'm not sure if your performance would be that great. The SuperMicro USAS L8i is PCI-express x8 gen1 so 2GB/s full duplex PCIe bandwidth and 2400MB/s SATA bandwidth.

http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm

There is an USAS2-L8e also, which is SATA/600. It works in OpenSolaris but not (yet) on FreeBSD. The regular USAS controller works fine on both; i use two of them. Price is little over $100, for 8 SATA devices. Cables were not included in my case; i had to buy two separate Mini-SAS -> 4x SATA cables.
 
sub.mesa said:
SuperMicro USAS-L8i is a 2x Mini-SAS controller that allows 8 SATA/300 devices to be connected as 'dumb' HBA; normal AHCI SATA controller without RAID.

That's most ideal to ZFS i believe. The Areca 1680iX is running with an internal port multiplier as far as i know; if you connect another port multiplier to that; well i'm not sure if your performance would be that great. The SuperMicro USAS L8i is PCI-express x8 gen1 so 2GB/s full duplex PCIe bandwidth and 2400MB/s SATA bandwidth.

http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm

There is an USAS2-L8e also, which is SATA/600. It works in OpenSolaris but not (yet) on FreeBSD. The regular USAS controller works fine on both; i use two of them. Price is little over $100, for 8 SATA devices. Cables were not included in my case; i had to buy two separate Mini-SAS -> 4x SATA cables.
Click to expand...
I know about the double expander problem but I thought the 1680x (not 1680ix) did not have an internal Expander?

I can't begin to build the system today as planned as I though the Norco 4220 supplied 2 SFF-8087 cables. I will have to wait until additional cables arrive as well as some low noise fans. I'm playing with Opensolaris right now although it does not support my SI 3132 cheap pcie (rosewell) controller card.
 
mikesm said:
Have you tried running lvm2 to span raid volumes? It's at least a 50% hit on streaming performance. Pretty much everyone on the mailing lists will verify this too - but it's the ability to recover from underlying volume errors that can take down the logical volume that's teh real killer.
Click to expand...

I don't even get why you would run lvm2?

Of course I definitely lean towards ZFS especially if software raid. The only reason I doon't use it for my main OS is the fact that I use my 40 TB storage box as a gaming and regular machine as I like the speed of DAS and I have to run linux on my main system. Any NAS system I will always go opensolaris first.

I would mainly advocate ZFS as IMHO your data is much more secure and safe/recoverable on it vs mdadm. Pulling the wrong drive because the /dev/sdX changed shouldn't ever happen IMHO unless you don't have a good setup.
 
nbat58 said:
Strange when I go to download 2010.03 it redirects me to version 2009.06 0906, what's all that about?

Anyone know what is the difference between 0906 and b134?
Click to expand...

OMG! 2010.03.... 3 months late... better late than never.

they reworked the CIFS client, the iSCSI client 2009.06 has terrible iSCSI performance, and a bunch of other things i have been awaiting 2010.03 for a wile now :D
 
Answering my own question the b134 is the 2010.03, I have downloaded the OpenSolaris 2010.03 but the booting time takes nearly 3 minutes, this makes OpenSolaris a no option for me as I switch my sistem off when not needed.

Back to the drawing board, looking at Nexenta next, will this be any better?
 
Another thing about ZFS is the disk space overhead. With my 18 TB volume after making it ZFS it lost about 500-600 GB of space compared to JFS.
 
nbat58 said:
Answering my own question the b134 is the 2010.03, I have downloaded the OpenSolaris 2010.03 but the booting time takes nearly 3 minutes, this makes OpenSolaris a no option for me as I switch my sistem off when not needed.

Back to the drawing board, looking at Nexenta next, will this be any better?
Click to expand...

Nexentra is still built on Solaris so i doubt it.

houkouonchi said:
Another thing about ZFS is the disk space overhead. With my 18 TB volume after making it ZFS it lost about 500-600 GB of space compared to JFS.
Click to expand...

Umm.... ZFS has no space overhead aside from any form of parity.. (raidz/z2/z3) So im not sure where you are losing the space.
 
Tau said:
Umm.... ZFS has no space overhead aside from any form of parity.. (raidz/z2/z3) So im not sure where you are losing the space.
Click to expand...

Well I don't know what to tell you other than that a lot seems to be missing:

Compare the total size:

ZFS:
Code: 
root@opensolaris: 11:22 AM :/data# df -k /data
Filesystem            kbytes    used   avail capacity  Mounted on
data                 17024716800 258872352 16765843815     2%    /data

JFS:
Code: 
root@sabayonx86-64: 11:22 AM :~# df -k /data2
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdd1            17577451416   2147912 17575303504   1% /data2

zpool list shows the raw capacity right?

Code: 
root@opensolaris: 11:25 AM :/data# zpool list data
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data   18.1T   278G  17.9T     1%  1.00x  ONLINE  -

Ok, i would expect it to be rounded to 18.2 but that seems about right for 20 trillion bytes (what 20x1 TB is):

Code: 
root@sabayonx86-64: 11:23 AM :~# echo | awk '{print 20000000000000/1024/1024/1024/1024}'
18.1899

Now minus two drives for parity:

Code: 
root@sabayonx86-64: 11:23 AM :~# echo | awk '{print 18000000000000/1024/1024/1024/1024}'
16.3709

Yet when running zfs list it also lists the amount of storage significantly smaller:

Code: 
root@opensolaris: 11:26 AM :/data# zfs list data
NAME   USED  AVAIL  REFER  MOUNTPOINT
data   247G  15.6T   247G  /data

With 247 GB used its harder to see the total size but empty it was 15.9 TB which is quite a bit away from the 16.4 it should have been.

Taking the df -k values JFS gives me a total volume size of:

Code: 
root@sabayonx86-64: 11:31 AM :~# echo | awk '{print 17577451416/1024/1024/1024}'
16.3703

and zfs is:

Code: 
root@sabayonx86-64: 11:31 AM :~# echo | awk '{print 17024716800/1024/1024/1024}'
15.8555

So tell me where that half TB went (between 500-600 GB) of space?
 
Tau said:
do you have duplication or something turned on?
Click to expand...

No:

Code: 
root@opensolaris: 02:45 PM :~/coreutils-8.5/src# zfs get all data
NAME  PROPERTY              VALUE                  SOURCE
data  type                  filesystem             -
data  creation              Sun May 30 14:32 2010  -
data  used                  164K                   -
data  available             15.9T                  -
data  referenced            56.0K                  -
data  compressratio         1.00x                  -
data  mounted               yes                    -
data  quota                 none                   default
data  reservation           none                   default
data  recordsize            128K                   default
data  mountpoint            /data                  default
data  sharenfs              off                    default
data  checksum              on                     default
data  compression           off                    default
data  atime                 on                     default
data  devices               on                     default
data  exec                  on                     default
data  setuid                on                     default
data  readonly              off                    default
data  zoned                 off                    default
data  snapdir               hidden                 default
data  aclmode               groupmask              default
data  aclinherit            restricted             default
data  canmount              on                     default
data  shareiscsi            off                    default
data  xattr                 on                     default
data  copies                1                      default
data  version               4                      -
data  utf8only              off                    -
data  normalization         none                   -
data  casesensitivity       sensitive              -
data  vscan                 off                    default
data  nbmand                off                    default
data  sharesmb              off                    default
data  refquota              none                   default
data  refreservation        none                   default
data  primarycache          all                    default
data  secondarycache        all                    default
data  usedbysnapshots       0                      -
data  usedbydataset         56.0K                  -
data  usedbychildren        108K                   -
data  usedbyrefreservation  0                      -
data  logbias               latency                default
data  dedup                 off                    default
data  mlslabel              none                   default
 
You must log in or register to reply here.
Back
Top