• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Replacing drive in FreeNAS ZFS array

F

Flapjack

2[H]4U
2FA
Joined
Apr 29, 2000
Messages
3,210
I had a drive fail in a 3-drive ZFS array a few days back. The zpool was degraded, so I ordered another drive. I shut down the system (since it's not hot-swappable) and replaced the bad drive.

There were two Samsung 1.5TBs, and a Seagate 1.5TB. Of course the POS Seagate failed, as I've had about 8 of them fail on me in the last five years. Not sure what I was thinking when I built the array, but oh well.

At any rate, the autonaming feature has effectively pissed me off. The remaining drives have shifted, and the new drive has a ridiculous long number for a name. I've disabled auto-generated UPT and UUID for new disks through the following loaders:

kern.geom.label.gptid.enable="0"
kern.geom.label.ufsid.enable="0"

I rebooted, hoping the drives would get back in their proper naming order, but of course, they didn't. It was so easy for them to rename themselves, but I can't seem to figure out how to name them back logically.

Here is the output from "zpool status -v" (vol1 is another drive being used for CIFS):
Code: 
[root@freenas] /dev# zpool status -v
  pool: vol0
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-4J
 scrub: none requested
config:

        NAME                      STATE     READ WRITE CKSUM
        vol0                      DEGRADED     0     0     0
          raidz1                  DEGRADED     0     0     0
            16887281796856902055  FAULTED      0     0     0  was /dev/ada0p2
            ada0p2                ONLINE       0     0     0
            ada1p2                ONLINE       0     0     0

errors: No known data errors

  pool: vol1
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        vol1        ONLINE       0     0     0
          ada2p2    ONLINE       0     0     0

errors: No known data errors

This, from "sysctl kern.disks"
Code: 
[root@freenas] /dev# sysctl kern.disks
kern.disks: ada2 ada1 ada0 da0

"camcontrol devlist" (which still shows the old Seagate drive, that I replaced with a Samsung matching the other two drives):

Code: 
[root@freenas] /dev# camcontrol devlist
<SAMSUNG HD154UI 1AG01118>         at scbus1 target 0 lun 0 (ada0,pass0)
<SAMSUNG HD154UI 1AG01118>         at scbus2 target 0 lun 0 (ada1,pass1)
<ST31500341AS CC1H>                at scbus3 target 0 lun 0 (ada2,pass2)
<Kingston DataTraveler 2.0 1.00>   at scbus6 target 0 lun 0 (da0,pass3)

"glabel status"
Code: 
[root@freenas] /dev# glabel status
          Name  Status  Components
 ufs/FreeNASs3     N/A  da0s3
 ufs/FreeNASs4     N/A  da0s4
ufs/FreeNASs1a     N/A  da0s1a

...and finally, a "gpart show":
Code: 
[root@freenas] /dev# gpart show
=>     63  7823214  da0  MBR  (3.7G)
       63  1930257    1  freebsd  [active]  (943M)
  1930320       63       - free -  (32K)
  1930383  1930257    2  freebsd  (943M)
  3860640     3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992  3918285       - free -  (1.9G)

=>      0  1930257  da0s1  BSD  (943M)
        0       16         - free -  (8.0K)
       16  1930241      1  !0  (943M)

=>        34  2930277101  ada0  GPT  (1.4T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  2926082703     2  freebsd-zfs  (1.4T)

=>        34  2930277101  ada1  GPT  (1.4T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  2926082703     2  freebsd-zfs  (1.4T)

=>        34  2930277101  ada2  GPT  (1.4T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  2926082703     2  freebsd-zfs  (1.4T)

Can anyone help?
 
I'm not sure I understand the sequence of events. Did the new drive fail to be usable from the get-go, or did this happen after you were trying to disable the UUID stuff. Why does it matter if the drives get renamed? ZFS can tell which drives are part of which pool... Also, I'm not understanding how a drive you have removed is still showing in camcontrol?
 
danswartz said:
I'm not sure I understand the sequence of events. Did the new drive fail to be usable from the get-go, or did this happen after you were trying to disable the UUID stuff. Why does it matter if the drives get renamed? ZFS can tell which drives are part of which pool... Also, I'm not understanding how a drive you have removed is still showing in camcontrol?
Click to expand...
It failed after about 6 months of use. I had VMs on it, but I was able to migrate them off (so I can resilver, wipe, etc... without losing anything).

The reason I'd like the drives to be in order is so that it makes sense! IMHO, it is absolutely retarded. As someone who's new to FreeNAS, I'm having a terrible time trying to run commands because I don't know which drive is which.

For example, if I go into the GUI, the first drive (the one that failed, which is in the first SATA slot on the board) show a blank for the name. If I click the "edit" button, it shows Name: ada0. The drives with ID 2 and 3 show ada0 and ada1, respectively, on the "View Disks" tab, but when I click "edit", their names are ada1 and ada2. It is total BS and bad programming to have done that. Why? There is no reason, and my OCD self wants to fix it for selfish reasons, in addition to helping it make sense in the future. I cannot rename them from the GUI, and I can't figure out how to do it command line.
 
Oh, and for the full sequence:

- Installed a 3-drive array roughly six months ago:
ada0 = Seagate 1.5TB
ada1 = Samsung 1.5TB
ada2 = Samsung 1.5TB

The Seagate failed a couple of days ago, so I bought a Samsung identical to the other two Samsungs, shut FreeNAS down, removed the Seagate (since it was dead) and put the new Samsung in.

I've tried a variety of "zpool replace" via commands via command line, but I'm obviously not getting the names right. I'm also not 100% sure there are no existing partitions on the new drive (was used), but I don't know how to tell. I've also tried a "replace" from the GUI, but there is nothing under the "Member Disk" pull down.
 
Well, not a bsd expert here, but if a dead/removed drive is still showing up, something is wrong and it isn't just the drive :( You might be better off creating a new pool from scratch.
 
danswartz said:
Well, not a bsd expert here, but if a dead/removed drive is still showing up, something is wrong and it isn't just the drive :( You might be better off creating a new pool from scratch.
Click to expand...
Bummer. What do you think could be wrong?
 
Here's what I did when I had similar problems with freenas last week.

*note: I'm not an expert, these are just the steps I took which fixed my issues*

First, you still need to replace the faulted drive. Find the new drives designation (ada0 or whatever)

Code: 
zpool replace vol0 16887281796856902055 ada0 (or, whatever its label is)
Wait for the resilver to complete. Then detach the old nonexistent drive.

Code: 
zpool detach vol0 16887281796856902055

Your array should then show up as ONLINE, instead of DEGRADED.

After that what you see in the gui won't match up with reality, so I'd install a clean copy of the new 8.2.0 release on a new usb key (just in case), and import the pool using the autodetect button in the gui. If you'd rather not do that, exporting the pool and re-importing it into your current install should fix the gui mismatch anyway.
 
MatDef said:
Here's what I did when I had similar problems with freenas last week.

*note: I'm not an expert, these are just the steps I took which fixed my issues*

First, you still need to replace the faulted drive. Find the new drives designation (ada0 or whatever)

Code: 
zpool replace vol0 16887281796856902055 ada0 (or, whatever its label is)
Wait for the resilver to complete. Then detach the old nonexistent drive.

Code: 
zpool detach vol0 16887281796856902055

Your array should then show up as ONLINE, instead of DEGRADED.
Click to expand...
I tried this, but I got the following error: "cannot replace 16887281796856902055 with ada0: no such pool or dataset"

After that what you see in the gui won't match up with reality, so I'd install a clean copy of the new 8.2.0 release on a new usb key (just in case), and import the pool using the autodetect button in the gui. If you'd rather not do that, exporting the pool and re-importing it into your current install should fix the gui mismatch anyway.
Click to expand...
I would definitely like to upgrade to 8.2.0. So I just need to load the files on the new USB key, install, then import the old pool? Wouldn't I need the files on the old USB key to do that... or is the data on the array itself?
 
ZFS metadata is on the pool devices. To see this, you can do 'zpool import' with no pool name, and it will tell you which (if any) pools are available to import and what devices they are on. Here is a thought: try this: 'zpool export vol0', then 'zpool import' and see what it says...
 
danswartz said:
ZFS metadata is on the pool devices. To see this, you can do 'zpool import' with no pool name, and it will tell you which (if any) pools are available to import and what devices they are on. Here is a thought: try this: 'zpool export vol0', then 'zpool import' and see what it says...
Click to expand...
It said:
"cannot unmount '/mnt/vol0': Device busy"
 
I hope you get your pool data back because its precisely these bizarre unexplained phenomena with ZFS that seem to crop up just often enough that I'm more than a little freaked out by it.
 
odditory said:
I hope you get your pool data back because its precisely these bizarre unexplained phenomena with ZFS that seem to crop up just often enough that I'm more than a little freaked out by it.
Click to expand...
Well, like I said, I was able to get all my Xen VMs off to local storage when I realized the drive failed. I may just blow the whole thing away and redo it. I'm a little hesitant, though, as I don't want to setup iSCSI and all that crap again.
 
No doubt there's always the nuclear "just format" option but you're probably like me and want to understand why something happened rather than just shrugging it off, so that you have a little more insight into whether or not it may happen again and avoiding the conditions that may have caused it.

And in fairness I have seen a lot of endorsements of FreeBSD + ZFS by major institutions so I dont actually believe ZFS is innately flawed, just that when weird crap happens whether by corruption or whatever and a pool doesn't mount with no rhyme or reason then your data is stuck behind that abstraction layer at least when we're talking raidz1/z2/z3.
 
odditory said:
No doubt there's always the nuclear "just format" option but you're probably like me and want to understand why something happened rather than just shrugging it off, so that you have a little more insight into whether or not it may happen again and avoiding the conditions that may have caused it.

And in fairness I have seen a lot of endorsements of FreeBSD + ZFS by major institutions so I dont actually believe ZFS is innately flawed, just that when weird crap happens whether by corruption or whatever and a pool doesn't mount with no rhyme or reason then your data is stuck behind that abstraction layer at least when we're talking raidz1/z2/z3.
Click to expand...
Missed your post earlier.

Yes, I could just blow it away, but I want to know why this is happening and how to fix it. Drive failures happen. I don't want to be going through this ever time it does.
 
You guys aren't going to believe this... I'm a dumbass. I went and looked and found the disk wasn't seated fully (HP Microserver with non-hot swap bays). I'm going through the steps of replacing the disk now.
 
My upgrade to a FreeNAS version 8.0.4 was when the USB flash drive ,where Freenas 8.0.1 resided, died due to power outrage [thanks to an shitty E-ton UPS].So I installed the new for the time version[8.0.4] and was able to import the pool [zpool1]via GUI with just one click. I was in surprise of trouble-free import via GUI,because at the old version [8.0.1] everything important must be done via CLI- the GUI of 8.0.1 was almost not usable.
Now,few months later,I still can't find single bug in the FreeNAS GUI 8.0.4.
Big improvement from those guys,I wonder how 8.2 works- after my current vacation that will be one of the first thing to do.
 
Ok, now that the resilvering is done, the pool still shows up as "DEGRADED", still shows the old drive, and still says "replacing"... even though the resilvering is done. I let it go all night.

Here's what is shows now:

Code: 
[root@freenas] ~# zpool status -v
  pool: vol0
 state: DEGRADED
 scrub: resilver completed after 1h8m with 0 errors on Thu Aug  2 00:28:31 2012
config:

        NAME                        STATE     READ WRITE CKSUM
        vol0                        DEGRADED     0     0     0
          raidz1                    DEGRADED     0     0     0
            replacing               DEGRADED     0     0     0
              16887281796856902055  OFFLINE      0     0     0  was /dev/ada0p2
              ada0                  ONLINE       0     0     0  127G resilvered
            ada1p2                  ONLINE       0     0     0
            ada2p2                  ONLINE       0     0     0

errors: No known data errors
 
zpool clear didn't seem to do anything, so I kept digging around in the gui... as most have said (through online searches) that most, if not all, the issues with failed disks could be resolved that way.

I finally figured out that going to "storage > view volumes > view status" gave me the option to "detach" the old disk. After that, I had to scrub the volume, then it finally said "healthy".

I learned alot through this whole experience. Thanks for all the help. I'm still overall impressed with FreeNAS, but this was definitely a PITA compared to commercial solutions (I work with NetApp day in and day out).
 
You must log in or register to reply here.
Back
Top