• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS Help

N3G4T1v3

n00b
Joined
Feb 3, 2011
Messages
9
Hey all

I'm rather stuck with a situation, and hope someone can help

Anyway, I built myself a media server with HP N36L microserver and 4 2TB EARX WD drives running in RaidZ

Originally I was using FreeBSD, but ended up messing something up and formatted it and moved to ubuntu (possibly not the best choice), mostly because not being really clued up with *nix in general, ubuntu was simpler to get around, although I have learnt a lot

Regardless, 2 of my drives started showing bad sectors, and this past weekend I decided to replace them, so I swapped out the 1st drive
To learn that using ZFSonLinux (Native ZFS) is some what flawed when it comes to this aspect
It has been working fine (well functioning, but the failing drives whee causing hassles), untill I tried to replace

using zpool replace command wouldn't work as it constantly return that the old drive I'm replacing, does not exist, which of coarse it does not, as it has been swapped out

I spent a good few hours trying, and researching, and all I turned up is that it is meant to be as easy as I had thought it would have been, but the command just doesn't work that way in ubuntu

So after getting frustrated, I downloaded Nas4Free, and made a live USB

And behold, importing the array, and running zpool replace, worked flawlessly, and 40 odd hours later (after loosing some data, due to the other drive being in bad shape as well) it completed

So all was well, I rebooted and loaded up ubuntu, and it read the repaired array
But it started resilvering again, so I left it to be

It completed again, but much quicker, like 8 hours, I thought all was ok
but even though it was done, I still get this:

Code:
            replacing-1                               DEGRADED     0     0     0
              3083943840068194838                     OFFLINE      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
              ata-ST2000DM001-1CH164_Z1E2T4HH         ONLINE       0     0     0

But thought nothing of it and assumed it would go away after a reboot

So I shutdown and pulled the old drive to go and swap it out, and got home, put in the new drive and booted up
To only notice that it is resilvering again, but not to the new drive, but to the other replacement drive
So again, I left it, and let it finish

Once it was done, which again was quickish, another 8 hours, of which my array was online and it was actually more stable with streaming and copying than before I replaced the drives, it has been functioning ok

So I rebooted and proceeded to boot up Nas4Free again, and import my array
Except now I cant, when I import the array directly, it gives and error, something like the variable is to large
So I ran zpool import -f, and the array imports, but says it is unavailable and says that the replaced drive is corrupted and obviasly the other drive is missing

So I reboot back into ubuntu and the array is available, and starts resilvering the replacement drive again

So now I'm stuck

I cant add the other drive to hope that it rebuilds, as ubuntu is a mess with ZFS (I'm really regretting the move) and the other OS says my array is corrupted
Code:
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Mon Jan 28 16:54:20 2013
    1,47T scanned out of 6,44T at 257M/s, 5h37m to go
    165M resilvered, 22,78% done
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          DEGRADED     0     0 1,61K
          raidz1-0                                    DEGRADED     0     0 3,23K
            3527413270148406706                       UNAVAIL      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516
            replacing-1                               DEGRADED     0     0     0
              3083943840068194838                     OFFLINE      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
              ata-ST2000DM001-1CH164_Z1E2T4HH         ONLINE       0     0     0  (resilvering)
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650  ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645  ONLINE       0     0     0

errors: 4163 data errors, use '-v' for a list

[The errors are noted, it doesn't seem to be loosing any thing more than on the rebuild, it just keeps noting the same stuff, maybe one or two other files are lost]

I'm at a loss
All I can think I can do now is to copy off what I can onto extra HDD space I have on my rig and other server and count my losses and destroy the array and create it again (probably on a differant OS)

So

With all that said, does anyone have any suggestions on what I should do

If I have to break my array down, what OS would you guys recommend I use instead, FreeBSD was cool, but it was a bit out of my league, spent to much time trying to get stuff to work or fix, I'm not sure what would be inbetween the ease of ubuntu and the harcoreness of BSD :p
 
First of all, FreeBSD is dead. They've got performance issues. They've got security issues. They can't fix them. They switched their own webservers to Linux after they got hacked on FreeBSD. I know others on this forum use it; they are all insane. You are insane for even considering FreeBSD here. Might as well implement ZFS on Haiku and randomly switch to that. In other words, there is NO reason to ever so much as consider FreeBSD for this purpose. People seem to think that just because ZFS was ported to FreeBSD that FreeBSD is now an equivalent option to Solaris. Well, it isn't.

Linux does have ongoing work for ZFS, both via FUSE and via a kernel module. Neither are at a point where they should be considered for use.

Run Solaris 11.1 or OpenIndiana or Illumian for ZFS. Nothing else. Use Napp-it for a GUI. I switched from Illumian to Solaris. But you might find Illumian easier, and it's still pretty good, and a much better ZFS option than any non-Solaris-based option. No non-Solaris-based option has anywhere NEAR the level of stability with ZFS that Solaris-based stuff does.

Note: I like Linux for non-ZFS stuff. I don't like FreeBSD at all anymore, for any purpose, and I've never really thought of it as being as good as Linux, anyway. If I were to use a BSD distribution, FreeBSD would be near the bottom of the list because of major security concerns. If I wanted to dump a bunch of features and performance by using BSD over Linux, then I'd at least require the better security NetBSD and OpenBSD have over FreeBSD.

P.S. Not exactly sure on the disk replacement part :/
 
Last edited:
Please give citations to your claims. I did find this: http://www.pcworld.com/article/2014744/hackers-crack-two-freebsd-project-app-dev-servers.html but it seems like SSH keys were stolen as a way to break into FreeBSD dev servers, not that FreeBSD was itself compromised.

First of all, FreeBSD is dead. They've got performance issues. They've got security issues. They can't fix them. They switched their own webservers to Linux after they got hacked on FreeBSD. I know others on this forum use it; they are all insane. You are insane for even considering FreeBSD here. Might as well implement ZFS on Haiku and randomly switch to that. In other words, there is NO reason to ever so much as consider FreeBSD for this purpose. People seem to think that just because ZFS was ported to FreeBSD that FreeBSD is now an equivalent option to Solaris. Well, it isn't.

Linux does have ongoing work for ZFS, both via FUSE and via a kernel module. Neither are at a point where they should be considered for use.

Run Solaris 11.1 or OpenIndiana or Illumian for ZFS. Nothing else. Use Napp-it for a GUI. I switched from Illumian to Solaris. But you might find Illumian easier, and it's still pretty good, and a much better ZFS option than any non-Solaris-based option. No non-Solaris-based option has anywhere NEAR the level of stability with ZFS that Solaris-based stuff does.

Note: I like Linux for non-ZFS stuff. I don't like FreeBSD at all anymore, for any purpose, and I've never really thought of it as being as good as Linux, anyway. If I were to use a BSD distribution, FreeBSD would be near the bottom of the list because of major security concerns. If I wanted to dump a bunch of features and performance by using BSD over Linux, then I'd at least require the better security NetBSD and OpenBSD have over FreeBSD.

P.S. Not exactly sure on the disk replacement part :/
 
Got my info from Phoronix, and I can't seem to find it now. There was a forum thread on it there also and it seems to have been deleted because of fighting between Linux/BSD fans :/. This does lower the value of my previous post significantly. Regardless though, I don't get this drive towards FreeBSD. It's just not better. If you want a fileserver then use something Solaris-based. If you want a non-file-server then use Linux which has more features, more hardware support, better performance, more community support (more devs, MUCH more frequent releases), etc.

It seems to me that FreeBSD is the new "hipster Linux" to you guys. Like all the "cool kids" have left Linux and gone onto FreeBSD.
 
Last edited:
I don't have strong feelings about FreeBSD but it's of interest to me since I run a NAS based off FreeBSD, reluctantly, as there is no native Linux ZFS support, and I didn't want to go down the Solaris/OI/Nexenta route.

I wonder about your knowledge of what's going on with FreeBSD when you make sweeping assertions like you did. I talked to my go-to friend for super-technical stuff like this, and he did not feel like FreeBSD was a big security risk or anything. He also noted--correctly--that FreeBSD is old school, so I have no idea where you are getting your "hipster Linux" thing from. If anything the drive has been AWAY from FreeBSD and towards Linux over the last several years, particularly the new darlings like Ubuntu.

It seems to me that some people misunderstood what happened with the FreeBSD dev server break-in, thinking that FreeBSD-the-OS got hacked when what actually happened was much more mundane. Perhaps that is what happened at the thread you referenced where people were claiming that FreeBSD got hacked or something.

If anyone else has anything to add about FreeBSD, I'm all ears.
 
Last edited:
Actually, someone who calls FreeBSD "old school" and claims users are leaving it for Linux probably has the outdated knowledge. Some users are indeed doing that. But in the past year or two, FreeBSD has been picking up again, because several features are being ported to FreeBSD, including some bad ideas Linux kernel devs have had. The migration these days goes both ways. There are still those making the correct choice to dump FreeBSD, but some of the new work is bringing new unfortunate souls into the dark side. And there are no cookies to be had.

Such as kernel mode setting (because GPU-specific code in your kernel is THE WAY TO GO, man!)

Such as FUSE (because, you know, it actually performs reas...wait, no it doesn't)

etc.

People here have been recommending it as a valid Solaris alternative for fileservers which it is NOT. Also, notice how several Linux distributions now have FreeBSD versions either available or in development. It IS an improvement to bring FreeBSD's kernel into use with a REAL package management system, but I still don't see the point.


Now if you think I seriously expect security issues in FreeBSD to NEVER get fixed, well, honestly that is not the case. I am sure they will be. I just.. don't care. Fixing issues and porting stuff from Linux is good and all, but they need to aim to be BETTER than Linux, not to be at parity. You may be looking at it as me needing to provide you with a suitable reason to NOT use FreeBSD, but I think FreeBSD needs to provide ME with a suitable reason to use it over Linux. And it has yet to come anywhere NEAR doing that.

Here's one thing about ZFS. Pretend you have an issue with it, like the OP does. Now you go to Google and look up info on ZFS. 90% of it is for Solaris, OpenSolaris, OpenIndiana, Illumos, Illumian, etc. Does the same advice work for FreeBSD? Maybe, maybe not. Are you screwed due to lack of advice? Maybe, maybe not...

What reason is there to use FreeBSD over Solaris for a fileserver? You hate Oracle and Larry Ellison? So do I - a LOT. Did I give them any money by using Solaris Express? No, I did not. Good enough for me.
 
Last edited:
Haha, guys, intense argument
Didn't realize that there was such a feud with FreeBSD, last I saw, people were praising it :p

First of all, FreeBSD is dead. They've got performance issues. They've got security issues. They can't fix them. They switched their own webservers to Linux after they got hacked on FreeBSD. I know others on this forum use it; they are all insane. You are insane for even considering FreeBSD here. Might as well implement ZFS on Haiku and randomly switch to that. In other words, there is NO reason to ever so much as consider FreeBSD for this purpose. People seem to think that just because ZFS was ported to FreeBSD that FreeBSD is now an equivalent option to Solaris. Well, it isn't.

Linux does have ongoing work for ZFS, both via FUSE and via a kernel module. Neither are at a point where they should be considered for use.

Run Solaris 11.1 or OpenIndiana or Illumian for ZFS. Nothing else. Use Napp-it for a GUI. I switched from Illumian to Solaris. But you might find Illumian easier, and it's still pretty good, and a much better ZFS option than any non-Solaris-based option. No non-Solaris-based option has anywhere NEAR the level of stability with ZFS that Solaris-based stuff does.

Note: I like Linux for non-ZFS stuff. I don't like FreeBSD at all anymore, for any purpose, and I've never really thought of it as being as good as Linux, anyway. If I were to use a BSD distribution, FreeBSD would be near the bottom of the list because of major security concerns. If I wanted to dump a bunch of features and performance by using BSD over Linux, then I'd at least require the better security NetBSD and OpenBSD have over FreeBSD.

P.S. Not exactly sure on the disk replacement part :/

Thanks for the advice
Well when I was looking at going with ZFS, the general opinion at the time seemed to point to FreeBSD, and I hadn't heard of OpenIndiana or Illumian at the time of consideration and Solaris didn't sound like what I was originally looking for, but I could have mistaken that

But FreeBSD was rather intense to try learn, and their forums are not newbie friendly, compiling everything was a luss

So with your three suggestions of File server OS, which has the a more friendly community to the less unix inclined :p
Also, do those OS's suggested all have package managers? or is it atleast reasonably simple to install new software?
But I'll try Illumian for now, hopefully it is possible to run it, or any of the others, as a live CD, so I can try repair my array, and toy around with the OS, before making the move
 
Last edited:
Illumian is an Illumos/OpenIndiana (both of these projects are OpenSolaris derivatives) with a Debian front-end. It is by far the easiest Solaris-based alternative to use. Downsides are in package management. Don't expect "some company" to update their proprietary package library nearly as often as any distro (be it of Linux or Solaris) using its "native" package management tool.

But how often to you need to add new software to a fileserver? I think Illumian is just fine for fileserver purposes. And being still OpenSolaris-based, there is a higher chance that ZFS-related information you find online will work with Illumian than there is with FreeBSD. Some of the advice will be the same between the two. Some will not. Hell, I wouldn't feel comfortable giving ZFS-on-FreeBSD advice, but I will sometimes help with Solaris-based ZFS (since that's what I use).

P.S. about the discussion above - I just use strong words in general. I've nothing personal against anyone for the use or recommendation of FreeBSD. I just don't really like FreeBSD much.
 
Last edited:
Awesome, thanks

As long as they have a fairly decent package manager, I'm happy
I don't install much on my fileserver, but now and them I like to try things, and it makes life easier to achieve them through a manager, even if the stuff isn't 100% up to the actual current version

I'm actually really excited to move to a new OS, although it is time consuming to learn a new Distro, I'm keen to have my server running like it should
As long as there is an active friendly community, and decent documentation, I'm all aboard

Thanks for the advice

Kind of bummed that no one has any advice about my current array issue, but I guess I've gotten myself into a rather bad situation, some what out of the norm
 
Kind of bummed that no one has any advice about my current array issue, but I guess I've gotten myself into a rather bad situation, some what out of the norm

There are people here that know more about ZFS than I do; hopefully for you some more of those guys will post after they get home from work or whatever.

You may find more people disagreeing with me about FreeBSD as well. Won't change my mind personally, but whatever.
 
Well, I never heard of L2ARC being buggy before. L2ARC is a core feature of ZFS. I never heard anyone say "Yeah, ZFS is great, just don't ever use L2ARC!" Until that massive Solaris failure L2ARC was on no radar as being problematic at all.

No non-Solaris-based option has anywhere NEAR the level of stability with ZFS that Solaris-based stuff does.

Solaris is FAR from perfect

You turn your head real quick.

They switched their own webservers to Linux after they got hacked on FreeBSD.

How is a hack through stolen ssh keys related to the OS at all? Did FreeBSD steal your lolly?
 
FreeBSD has a vastly superior network stack. It is incredibly stable, if you're doing something standard and it worked once. The design philosophy is also more to some peoples liking, rather than anyone can do anything any time, with one dictator deciding the top level changes.

That said, I only use it is specific use cases... mostly router/firewall related (pfSense), or other basic network services (dns/http/etc). I wouldn't probably run a serious production server on FreeBSD+ZFS (yet), but I also wouldn't fear it if it's properly backed up and I had time to fix things if they crop up without hordes of people with pitchforks assembling.
 
FreeBSD runs up to ZFS v28 and I don't think it's that big of a deal if Solaris is on a supposedly more advanced version. v28 is "good enough" for me and probably for the vast majority of home users just looking to set up a file server; in fact, a lot of people think ZFS of any flavor is overkill as it is, let alone the latest version of ZFS. So the whole if-someone-advises-x-for-Solaris-it-might-not-work-for-FreeBSD thing is a nonissue imho.

However, the talk about FreeBSD security holes got my attention. I won't tolerate security holes, and if this is a real problem, I want to know about it. So far it looks like FreeBSD itself didn't have security issues, just that someone stole SSH keys and broke into a server. That could happen with any OS so I'm waiting to hear more... were there in fact holes discovered in FreeBSD, and if so, what is the current status?
 
FreeBSD runs up to ZFS v28 and I don't think it's that big of a deal if Solaris is on a supposedly more advanced version. v28 is "good enough" for me and probably for the vast majority of home users just looking to set up a file server; in fact, a lot of people think ZFS of any flavor is overkill as it is, let alone the latest version of ZFS. So the whole if-someone-advises-x-for-Solaris-it-might-not-work-for-FreeBSD thing is a nonissue imho.

Try not to view ZFS versions as a must-have-the-latest-bells-and-whistles. AFAIK v28 was the last version to be available with source code. From there on, you're in Oracle's vendor-lockin territory. See v28 as more flexible because you have several systems available that can work with it if something goes so wrong that you might want to try another OS.

Another thing: last time I checked, Solaris' encryption support didn't really support AES-NI. It was advertised as working but didn't actually work. With FreeBSD's full-disk encryption GELI, you're getting actual hardware acceleration. On a single Xeon E5-2609 core with 2.4GHz I get above 500MB/s throughput with AES-128-CBC. Multiple disks multithread real well.

Seeing how really horrible Oracle's software has become*, I'll take open-source any day. Maybe I'm biased but I find BSD documentation to be better than anything else out there. I never really got into the OpenSolaris derivates. They just feel obscure to me.


* When their Java plugin isn't busy getting pwned, it tries to shove adware down your throat at every possibility. "Oh and tough luck about that ZFS bug. Just recreate your pool, and unless you pay us, happy waiting for a fix."
 
Try not to view ZFS versions as a must-have-the-latest-bells-and-whistles. AFAIK v28 was the last version to be available with source code. From there on, you're in Oracle's vendor-lockin territory. See v28 as more flexible because you have several systems available that can work with it if something goes so wrong that you might want to try another OS.

I'm not sure where you are getting that from and in fact we both agree, I think. I stated that that I think v28 is "good enough" for me and probably most other home file server users. The fact that it's possible to move from one v28 system to another is a nice bonus but one I doubt I'll use. What is more of a problem for me re: Solaris is that Oracle is involved, and I simply can't bring myself to trust Oracle from changing the Solaris ecosystem in a negative way. So I simply avoid it.

That said, if btrfs were ready and native to Linux and more mature (I am not about to use the currently-available workarounds on any important data), I'd use that in a heartbeat.
 
Can you post the current output from "zpool status" on Ubuntu, and list the exact commands you used to attempt the replacement. Before and after "zpool status" outputs would be handy too!


While personally I don't think switching OS in the middle of a problem is a good idea, did you ensure that you exported the pool first in one OS before importing it to the other?


BTW, I tested quite a few different failure scenarios before switching to Ubuntu with ZFS-on-Linux, and didn't encounter any issues replacing disks.
 
Last edited:
Can you post the current output from "zpool status" on Ubuntu, and list the exact commands you used to attempt the replacement. Before and after "zpool status" outputs would be handy too!


While personally I don't think switching OS in the middle of a problem is a good idea, did you ensure that you exported the pool first in one OS before importing it to the other?


BTW, I tested quite a few different failure scenarios before switching to Ubuntu with ZFS-on-Linux, and didn't encounter any issues replacing disks.

Code:
$ sudo zpool status
  pool: tank
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
 scan: scrub in progress since Wed Jan 30 07:11:54 2013
    6,26T scanned out of 6,44T at 186M/s, 0h16m to go
    0 repaired, 97,24% done
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          DEGRADED     0     0 4,90K
          raidz1-0                                    DEGRADED     0     0 9,80K
            3527413270148406706                       UNAVAIL      0     0     0                                                     was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516
            replacing-1                               DEGRADED     0     0     0
              3083943840068194838                     OFFLINE      0     0     0                                                     was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
              ata-ST2000DM001-1CH164_Z1E2T4HH         ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650  ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645  ONLINE       0     0     0

errors: 5020 data errors, use '-v' for a list

I set it to scrub, as you can see, it is almost done

Before I even replaced my drives, my zpool status looked like this (I dont have an exact copy of it, but I know more or less what it was returning)

Code:
$ sudo zpool status
  pool: tank
 state: ONLINE
status: 
 scan: scrub in progress since Wed Jan 30 07:11:54 2013
    6,26T scanned out of 6,44T at 186M/s, 0h16m to go
    0 repaired, 97,24% done
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          ONLINE     0     0     0
          raidz1-0                                    ONLINE     0     0 	 0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516  ONLINE     0     0     0                                                     
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385  ONLINE	 0	   0	 0	
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650  ONLINE     0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645  ONLINE     0     0     0

So drive names I have there were generated by ubuntu, when I imported my array from freeBSD (I'm assuming that this could be the reason for some of my issues that I have been having with the replace command)

So to replace I tried

Code:
#zpool replace tank ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385 /dev/sdb

and it would return something like, ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385 is invalid or does not exsist
(or something similar)

I then, after some digging, realized that those are short names and found its path

so it tried
Code:
#zpool replace tank /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385 /dev/sdb

again it returned that it was the incorrect name

I exported the array and loaded up nas4free (which, for what I understand is based on FreeBSD 9)
Importing it I got something like this

Code:
$ sudo zpool status
  pool: tank
 state: Degraded

config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          ONLINE     0     0     0
          raidz1-0                                    ONLINE     0     0 	 0
            disk/label1								  ONLINE     0     0     0                                                    
            3083943840068194838						  UNAVAIL	 0	   0	 0
            disk/label3								  ONLINE     0     0     0
            disk/label4  							  ONLINE     0     0     0

So I ran
Code:
zpool replace tank 3083943840068194838 /dev/ada2

and this worked and began replacing the drive

after it was done, I'm pretty sure I exported the array again and booted into ubuntu to import it

it then started resilvering the drive again, no idea why

I then replaced the 2nd drive, and booting up again, the drive started resilvering, yet again
When it was done, I exported and tried to import to nas4free, but it gives and error saying the variable is to large, or something like that, going zpool import -f, imports the array, but shows that the array is unavailable, and the replaced drive is corrupt

booting up ubuntu, the drive starts to resilver again on import


So now when I try replace it with
#zpool replace tank 3527413270148406706 /dev/sda
it would return that I require a a letter or something before 3527413270148406706

CHANGE OF EVENTS
Lol

Ok, So the scrub complete, so I ran
#zpool replace tank 3527413270148406706 /dev/sda
It accepted the command, I'm confused

Code:
$ sudo zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Wed Jan 30 17:23:13 2013
    11,3G scanned out of 6,44T at 66,1M/s, 28h20m to go
    2,83G resilvered, 0,17% done
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          DEGRADED     0     0 5,09K
          raidz1-0                                    DEGRADED     0     0 10,2K
            replacing-0                               UNAVAIL      0     0     0
              3527413270148406706                     UNAVAIL      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516
              sda                                     ONLINE       0     0     0  (resilvering)
            replacing-1                               DEGRADED     0     0     0
              3083943840068194838                     OFFLINE      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
              ata-ST2000DM001-1CH164_Z1E2T4HH         ONLINE       0     0     0  (resilvering)
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650  ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645  ONLINE       0     0     0

errors: 2700 data errors, use '-v' for a list
(The errors are from the scrub, the still the same lost files from the original resilver, due to the other bad HDD)

Well we will see in 28 or so hours

I'm still confused why now both drives show resilvering?
 
Me too, but I guess we'll see when the resilver process ends...


As to the disk replacement itself.....

If you start with:


NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645 ONLINE 0 0 0


and you then shutdown and replace say, ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
You'll get this when you reboot

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385 UNAVAIL 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645 ONLINE 0 0 0

If you attempt

#zpool replace (-f) tank ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385....blah...blah

it will fail, as this device entry no longer exists under /dev/disk/by-id.

The solution is to simply export and re-import the pool once the system has rebooted (or explicitly export before shutting down, and hence explicitly import after reboot).
This will "kickstart" ZFS into noticing that a disk is missing.
This situation arises because at boot-time ZFS relies on the zpool.cache file for the pool configuration information, rather than reading the info from disk.
This zpool.cache file is the zpool config in use when the system went down - however changing a disk when the system is off invalidates the cached config for that pool.
The export/reimport causes ZFS to delete the entry for that pool from the zpool.cache file, and upon import, read the real config from disk and create a new entry in the zpool.cache file.



Assuming you reimport using unique id naming (ie zpool import -d /dev/disk/by-id tank)

You'll then get


NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516 ONLINE 0 0 0
6543328653498745623 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650 ONLINE 0 0 0
ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645 ONLINE 0 0 0


after which you can then replace the disk using

#zpool replace tank 6543328653498745623 /dev/disk/by-id/<new disk name>
or
#zpool replace tank 6543328653498745623 /dev/sdX


The number "6543328653498745623" is just randomly generated by ZFS.


This is common to ZFS, but the disk naming conventions are unique to each OS.
Hence it's useful to fully understand the disk naming convention of whichever OS you choose, and how that particular OS's ZFS implementation uses this information.
It's also useful to understand how your chosen OS labels/partitions disks, and how that OS's ZFS implentation does it by default - they aren't all the same!


The apparent success of NAS4Free with the zpool replace command, where Ubuntu apparently failed, is almost certainly simply because as NAS4Free had no entry in it's zpool cache file for it, and so was forced to read the data from disk at boot time.
Hence that automatically put the pool into a state where the device could be replaced.
The device replacement issue you encountered can occur on BSD and Solaris systems too!


Hopefully it'll all come back good after the resilver!!
 
Thanks for the clearing that up, it makes more sense now

Ok, but now I'm still getting this"

Code:
  pool: tank
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
 scan: scrub repaired 0 in 10h7m with 2506 errors on Fri Feb  1 17:04:06 2013
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          DEGRADED     0     0 4,89K
          raidz1-0                                    DEGRADED     0     0 9,79K
            replacing-0                               DEGRADED     0     0     0
              3527413270148406706                     UNAVAIL      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516
              sda                                     ONLINE       0     0     0
            replacing-1                               DEGRADED     0     0     4
              3083943840068194838                     OFFLINE      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
              ata-ST2000DM001-1CH164_Z1E2T4HH         ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650  ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645  ONLINE       0     0     0

errors: 2506 data errors, use '-v' for a list

after a reboot, it will start resivering again, on both drives :/

I ran a scrub, but it passed through fine, not finding any additional errors
How do I now go by telling the array that I accept the erorrs, and dismiss them, so it will stop trying to fix them? (I'm assuming that is the current issue)
 
Yeah, run [ZFS] that piece of shit. http://simplex.swordsaint.net/?p=199
Uhm you know that every software has bugs, right? Bugs are crawling all over Linux and Windows too, if you did not know. And other filesystems has bugs too.

Regarding Solaris, the same blogger that you linked too, says in a later post:
http://simplex.swordsaint.net/?p=470#more-470
"I also believe that Solaris today is one of the best, most reliable UNIX today, still when I see a trend like this, I wish Oracle would put more effort behind game changer technologies like ZFS".

Regarding OpenSolaris open sourcing the code:
http://simplex.swordsaint.net/?p=17
"Also you had a chance to silently give respect to Sun&#8217;s kernel developers how clean and simple a code can be even if the result is a complex monster like SunOS."

It looks as if he praises both Solaris and ZFS, dont you think?





On the other hand, especially Linux has lots of problem with code quality. Linus Torvalds says Linux is bloated and huge:
http://www.theregister.co.uk/2009/09/22/linus_torvalds_linux_bloated_huge/
"Citing an internal INTEL corp study that tracked kernel releases, Bottomley said Linux performance had dropped about two per centage points at every release, for a cumulative drop of about 12 per cent over the last ten releases. "Is this a problem?" he asked. "We're getting bloated and huge. Yes, it's a problem," said Torvalds."


Linux kernel developer Andrew Morton says the code quality is declining:
http://lwn.net/Articles/285088/
"I used to think [code quality] was in decline, and I think that I might think that it still is. I see so many regressions which we never fix....it would help if people's patches were less buggy."


http://kerneltrap.org/Linux/Active_Merge_Windows
"The [linux source code] tree breaks every day, and it's becomming an extremely non-fun environment to work in...We need to slow down the merging, we need to review things more, we need people to test their f--king changes!"


http://milek.blogspot.se/2010/12/linux-osync-and-write-barriers.html
"This is really scary. I wonder how many developers knew about it especially when coding for Linux when data safety was paramount. Sometimes it feels that some Linux developers are coding to win benchmarks and do not necessarily care about data safety, correctness and standards like POSIX. What is even worse is that some of them don't even bother to tell you about it in official documentation"


Ted Tso, ext4 developer, says that Linux devs cheat and produce bad code, just to win performance benchmarks:
http://phoronix.com/forums/showthread.php?36507-Large-HDD-SSD-Linux-2.6.38-File-System-Comparison&p=181904#post181904
"In the case of reiserfs, Chris Mason submitted a patch 4 years ago to turn on barriers by default, but Hans Reiser vetoed it. Apparently, to Hans, winning the benchmark demolition derby was more important than his user's data. (It's a sad fact that sometimes the desire to win benchmark competition will cause developers to cheat, sometimes at the expense of their users.)...We tried to get the default changed in ext3, but it was overruled by Andrew Morton, on the grounds that it would represent a big performance loss, and he didn't think the corruption happened all that often (!!!!!) --- despite the fact that Chris Mason had developed a python program that would reliably corrupt an ext3 file system if you ran it and then pulled the power plug "


OpenBSD developer Theo de Radt says the Linux code is bad
http://www.forbes.com/2005/06/16/linux-bsd-unix-cz_dl_0616theo.html
"It's terrible," De Raadt says. "Everyone is using [Linux], and they don't realize how bad it is. And the Linux people will just stick with it and add to it rather than stepping back and saying, 'This is garbage and we should fix it.'"


In an interview with German newspaper Zeit Online, Torvalds recently stated that Linux has become "too complex" and he was concerned that developers would not be able to find their way through the software anymore. He complained that even subsystems have become very complex and he told the publication that he is "afraid of the day" when there will be an error that "cannot be evaluated anymore."
http://www.tomshardware.com/news/Linux-Linus-Torvalds-kernel-too-complex-code,14495.html




Actually, I have not heard similar complaints on FreeBSD code quality. I would chose FreeBSD over Linux anyday. FreeBSD has a very good reputation among serious Unix sysadmins, whilst Linux has very bad reputation. Just talk with them. They say they Linux is unstable and servers need to be rebooted every once in a while.
 
Thanks for the clearing that up, it makes more sense now

Ok, but now I'm still getting this"

Code:
  pool: tank
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
 scan: scrub repaired 0 in 10h7m with 2506 errors on Fri Feb  1 17:04:06 2013
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          DEGRADED     0     0 4,89K
          raidz1-0                                    DEGRADED     0     0 9,79K
            replacing-0                               DEGRADED     0     0     0
              3527413270148406706                     UNAVAIL      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9725516
              sda                                     ONLINE       0     0     0
            replacing-1                               DEGRADED     0     0     4
              3083943840068194838                     OFFLINE      0     0     0  was /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA9745385
              ata-ST2000DM001-1CH164_Z1E2T4HH         ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9718650  ONLINE       0     0     0
            ata-WDC_WD20EARX-00PASB0_WD-WCAZA9748645  ONLINE       0     0     0

errors: 2506 data errors, use '-v' for a list

after a reboot, it will start resivering again, on both drives :/

I ran a scrub, but it passed through fine, not finding any additional errors
How do I now go by telling the array that I accept the erorrs, and dismiss them, so it will stop trying to fix them? (I'm assuming that is the current issue)


Hmm, your pool appears to be in an inconsistent state now.
Which OS did you perform the disk replacements under?
If it was NAS4Free you may have to boot that and see what what state the pool is in under that OS - don't forget to export the pool from Ubuntu first!

Failing that, if your data is accessible and seems OK, you could try a "zpool clear tank", though I suspect in this case, it may not have the desired effect!

You could also try a "zpool export tank", followed by a "zpool import -d /dev tank", just in case the issue is to do with disk naming.
 
Hmm, your pool appears to be in an inconsistent state now.
Which OS did you perform the disk replacements under?
If it was NAS4Free you may have to boot that and see what what state the pool is in under that OS - don't forget to export the pool from Ubuntu first!

Failing that, if your data is accessible and seems OK, you could try a "zpool clear tank", though I suspect in this case, it may not have the desired effect!

You could also try a "zpool export tank", followed by a "zpool import -d /dev tank", just in case the issue is to do with disk naming.

One drive I did the replacement under Nas4Free, the 2nd under ubuntu (as I couldn't import the array into Nas4Free, and managed to get ubuntu to accept the replace command, somehow)

I tried zpool clear tank, and the pool started resivering the 2 replaced drive "-_-, I'm at a loss with that, really
Scrubbing doesn't do anything though, it doesn't trigger any repairs or resilvering, but why in the world would the clear command trigger it? it's never in the past

Could you explain what "zpool import -d /dev tank" does? what does the -d command do?
I'll have to give it a try later
 
Back
Top