Anyone using ZFS on their fileserver?

C

Cryptic1911

Limp Gawd
Joined
Jul 6, 2004
Messages
226
I have a system with 8x500gb drives that I'm thinking of using with freeBSD and ZFS in a raid-z setup.. does anyone have any concerns that I should look into before I do this? I've messed around with it with a couple drives to test, but I'm waiting for a 4x sata -> sas bridge board to come in before I can use all 8 drives (should be in tomorrow)

anyways, here's the specs:

freeBSD 7.0 x64
Q6600 4x2.4ghz
4gb ram
Giigabyte n680sli-dq6 board
Lsi 8344elp 128mb 8x sas raid card (using 4x sas -> sata bridge boards & multilane sas cables to run drives in external case)
8x 500gb Seagate 7200.10 sata
Intel pro 1000mt dual gigabit nic
 
only concern at the moment is that it's impossible to expand raid-z arrays. They're planning to add that ability in near future.
 
from what I've read, you might want to put /boot on a usb disk,thumbdrive, or cd or
something, so zfs can use all of the disk; that way it is quicker. see threads at
freebsd-questions (google groups: ) lucky freebsd-questions digest zpool
......................................................^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ that search.
People have been asking and answering questions like yours
 
Looks like i didnt need the differentl 4x sata -> sas bridge after all.. Ends up I had a loose cable so it only showed 6 drives instead of all 8

anyways, I got all 8 drives fired up and created a 3.62TB raid-z array. I originally thought you could expand them, but after thinking about it, the most likely thing I would do is to do an in place upgrade to 1tb drives once the prices come down..So I should be fine on that front

Now all I gotta do is setup samba / nfs and appletalk and copy my data over
 
ok, so that didn't go so well.

I was all excited about ZFS since it sounded like it was going to be the new wave, but I found it quite unstable. I did some searching online, and other people had the same exact issues I had. Here's one guy that kinda sums up his experience, and I will quote the part that pertained to me: http://www.mouldy.org/zfs-the-final-straw

"FreeBSD 7 (talking about ZFS)
leaks memory and crashes if you use it a lot, but not as badly as Linux. Still has the precious-data-disappearing bug. Barfs if the drive arrays aren't just right when you boot up. An array that I created and copied data to just did not exist the next time I booted the machine."


here's what how it went down for me:

I had setup a raid-z zpool with 8 drives attached to a hardware raid controller, setup samba and then started copying ~50gb over. It got about 6gb copied and then my Mac Pro sorta half locked up. It basically locked up my file manager and I couldnt shut the machine down properly. I hopped onto the console of the FreeBSD machine and it was spewing out mfi0 timeout messages, and ANYTHING I tried to do with the zfs filesystem would hang.. even an "ls". I ended up issuing a shutdown command, which also hung. I finally powered down the machine, and fired it back up. After it booted, I checked zpool list and zpool status, everything was still listed as online and working properly, but my data was gone.. and I know it had copied about 6gb of data over


so... yeah, I'm gonna pass on ZFS for now because of the fact that it hosed both my Mac and my FreeBSD machine requiring hard reboots, and the fact that it lost the only data I had tried copying over.
 
actually, I should take that back.. Its probably only 3rd party ZFS releases on NON Solaris OS's. This guy tested it in FUSE on Linux and native in FreeBSD.. and I also had FreeBSD issues.

maybe I'll give Opensolaris a shot and see what happens.. depends how motivated I am when I get home. The only issue is that I have no Solaris experience so it'll be a trip for sure
 
I am using ZFS for my fileserver. It's running OpenSolaris, and I haven't had any major problems with it---certainly no data loss problems. The people I've seen who have data loss problems are very vocal about their losses---no surprise there!---but they're generally trying to do difficult things in a weird way. For example, in the blog you link, the guy is trying to boot from USB. That's not supported yet, that I know of. If you have supported controllers (Areca, AHCI, Sil3***, Marvell, others) it'll probably work well for you.

If you run into problems with OpenSolaris, post about them and I'll see if I can help.
 
well from posts similar to the one in the freebsd-questions digest-list volume 223 issue 14
and similar 220-224* numbered digests, zfs is doable on freebsd after pretuning.
adequate memory, setup a certain way, etc. It would take me a good 4 hours or so
reading everything before I'd try to set one up. You crash *might* have been due to the
memory-hungriness of zfs compared to other FS especially for that specific file copy.
 
Cryptic1911 said:
so... yeah, I'm gonna pass on ZFS for now because of the fact that it hosed both my Mac and my FreeBSD machine requiring hard reboots, and the fact that it lost the only data I had tried copying over.
Click to expand...
Try it with Solaris / OpenSolaris.

ZFS is just newly ported to FreeBSD so there may be issues, but I know in Solaris / OpenSolaris it's very solid and stable.
 
Ok, I got OpenSolaris 2008.05 installed. It detected everything except my lsi 8344elp sas raid card on the live cd. After I got it installed to disk, I downloaded the Solaris driver for my raid card and installed it, and it found the 8 drives immediately and I was able to create a zpool out of them.

I dont know of any other way to test disk throughput at the moment, but I setup FTP and transferred some files: 700mb & 7gb. it transferred the 700mb in like 7 seconds / 92.95MB/s, but the 7gb file took like 1.5-2mins. I watched it go, and it seemed to send in spurts, then pause, so the throughput was ~60MB/s on that one.

I noticed that its a bit cpu intensive, but overall everything seems to be going pretty good. I just have to find some good howto's or a guide or something to get me on the right track with Solaris
 
Install CIFS and then you can use it in Windows as a network drive. Or turn on NFS if you have a Linux box: "zfs set sharenfs=on mypool/myfs".
 
unhappy_mage said:
Install CIFS and then you can use it in Windows as a network drive. Or turn on NFS if you have a Linux box: "zfs set sharenfs=on mypool/myfs".
Click to expand...

You may have to reboot after installing the packages. On the one I set up last night, it didn't have the smb/server service until I rebooted. Once I rebooted, it worked fine, and I saw transfer rates around 100MB/s over gigE. Made me happy :p
 
thanks unhappy_mage and rogue_jedi.. I ssh'ed back to my house from work and installed CIFS, had to reboot, and then enabled the service and joined my workgroup

got any good links with general knowledge stuff that I should know for Opensolaris? I'm gonna need to read up on things since this is a bit different than linux
 
Generally speaking, Google is a good reference. I would subscribe to the indiana-discuss mailing list (if you don't mind a little more email traffic every day) and watch for how to do upgrades. OTOH, if you're content with the meager set of packages that are installed by default, you're pretty much good to go ;) I'd consider setting up automatic ZFS snapshots for your home directory, and possibly your data directory as well.

Other than that... look on mailing lists for things you're interested in doing.
 
drizzt81 said:
someday I want a solaris ZFS box :)
Click to expand...

I have Solaris boxes. They all went back to proper hardware RAID on the SAN after two weeks of testing, and two statements from Sun that "ZFS is not ready for production environments" and "Sun will not support ZFS in production environments."

FreeBSD's ZFS is superior to OpenSolaris' for stability, and that should scare the absolute crap out of you. You could not pay me to use ZFS after testing it. It decimates the CPU because of how badly written the design is, and just wait till you find out what it does with your memory. It's not just the overhead, but just what absurd amounts of uncommitted data it loves to keep in there.
 
AreEss said:
I have Solaris boxes. They all went back to proper hardware RAID on the SAN after two weeks of testing, and two statements from Sun that "ZFS is not ready for production environments" and "Sun will not support ZFS in production environments."
Click to expand...
When was this (i.e., which Solaris), and what do you mean by "support"? Frankly, I'd be surprised if Sun "supports" UFS.
AreEss said:
FreeBSD's ZFS is superior to OpenSolaris' for stability, and that should scare the absolute crap out of you. You could not pay me to use ZFS after testing it. It decimates the CPU because of how badly written the design is, and just wait till you find out what it does with your memory. It's not just the overhead, but just what absurd amounts of uncommitted data it loves to keep in there.
Click to expand...
What stability issues have you seen? On what version of Solaris? What tests did you run that showed problems? What do you mean by "decimating" the CPU? Have you looked into ARC cache limits, since large amounts of data in memory are problematic?

I understand being gun-shy if you've had problems with ZFS in the past, but to be frank I haven't seen any issues at all with it. My applications are a little less critical than yours, I think, but 70 MB/s from a single disk over NFS is perfectly acceptable performance for most home uses. I've never seen any "stability" issues in terms of destroyed filesystems, lost files, or that kind of thing, and the only time I've paniced a box is when I rebooted the external array without exporting the pool.
 
You must log in or register to reply here.
Back
Top