• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

zfs trying to replace drive

Zedicus

Zedicus

[H]ard|Gawd
2FA
Joined
Nov 2, 2010
Messages
1,472
ive got a 6 drive raidz2 that was seagate based and some of the drives started acting up. ive got a few of them replaced and figured i would just work through replacing them all when this started happening. 1 drive (da3) gives an error when i try to detach it "disk offline failed - no valid replicas" so i went ahead and issued replace and the new drive (drives now) come up underneath the failing drive. what did i screw up and is this salvageable or do i need to build a new raidz2 and copy the data before i hose it completely? i have some partial backups and really nothing on it is irreplaceable. but the time involved makes me want to salvage or attempt to copy the data over if i can.


i will post the zfs pool status momentarily
 
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed May 20 15:30:02 2015
340G scanned out of 8.69T at 253M/s, 9h36m to go
226G resilvered, 3.82% done
config:

NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 25
raidz2-0 ONLINE 0 0 50
gptid/2a6da0e6-fa8d-11e4-bfd3-32a10598571e ONLINE 0 0 0 (resilvering)
gptid/8a8d757f-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
gptid/8b67d5b2-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
gptid/8c45976e-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
replacing-4 ONLINE 0 0 0
... gptid/8d24b451-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
... gptid/908fc82f-fd00-11e4-bfd3-32a10598571e ONLINE 0 0 0 (resilvering)
... gptid/fd930d38-ff2e-11e4-a38e-32a10598571e ONLINE 0 0 0 (resilvering)
gptid/d1bb59ac-fdbb-11e4-a38e-32a10598571e ONLINE 0 0 0 (resilvering)

errors: 5 data errors, use '-v' for a list
[root@freenas] ~#
 
yes there are checksum errors, i have scrubbed till blue in the face and the drive i have not been able to detach is part of the problem.
 
Just 1.

why its showinglike that i do not know, and why under 'replacing'' it shows 3 tabbed in drives, as if they are all associated together, i just do not understand. ive replaced single drives via the standard 'ofline bad drive' 'replace using new drive' 'detach old drive' and never had this issue... obviously i screwed something up this time.
 
Last edited:
its 'completed' but i get this mess.


scan: resilvered 5.79T in 9h33m with 5 errors on Thu May 21 01:03:22 2015
config:

NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 30
raidz2-0 ONLINE 0 0 60
gptid/2a6da0e6-fa8d-11e4-bfd3-32a10598571e ONLINE 0 0 0
gptid/8a8d757f-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
gptid/8b67d5b2-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
gptid/8c45976e-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
replacing-4 ONLINE 0 0 0
... gptid/8d24b451-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
... gptid/908fc82f-fd00-11e4-bfd3-32a10598571e ONLINE 0 0 0
... gptid/fd930d38-ff2e-11e4-a38e-32a10598571e ONLINE 0 0 0
gptid/d1bb59ac-fdbb-11e4-a38e-32a10598571e ONLINE 0 0 0
 
heres what happens when i offline one of the disks, its almost like its its own pool...

pool: zfs
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu May 21 06:15:30 2015
1.18G scanned out of 8.69T at 48.3M/s, 52h21m to go
594M resilvered, 0.01% done
config:

NAME STATE READ WRITE CKSUM
zfs DEGRADED 0 0 30
raidz2-0 DEGRADED 0 0 60
gptid/2a6da0e6-fa8d-11e4-bfd3-32a10598571e ONLINE 0 0 0 (resilvering)
gptid/8a8d757f-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
gptid/8b67d5b2-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
gptid/8c45976e-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
replacing-4 DEGRADED 0 0 0
... gptid/8d24b451-2ca9-11e4-865b-001b78595db4 ONLINE 0 0 0
... 14409482915547828078 OFFLINE 0 0 0 was /dev/gptid/908fc82f-fd00-11e4-bfd3-32a10598571e
... gptid/fd930d38-ff2e-11e4-a38e-32a10598571e ONLINE 0 0 0 (resilvering)
gptid/d1bb59ac-fdbb-11e4-a38e-32a10598571e ONLINE 0 0 0 (resilvering)
 
Can you put it in
Code: 
 tags so that it keeps the formatting? It's hard to read like that, the GPT id's are duplicated in the lower section from the top drive that was also showing as resilvering which i've never seen personally.

Can you also get the gptid to daX mapping as well so we can see which drive is which

Is this plain FreeBSD or FreeNAS?
 
Code: 
[root@freenas] ~# gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 25165823
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: ada0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r1w0e1
   attrib: active
   rawtype: 165
   length: 988291584
   offset: 32256
   type: freebsd
   index: 1
   end: 1930319
   start: 63
2. Name: ada0s2
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 988356096
   Mode: r0w0e0
   rawtype: 165
   length: 988291584
   offset: 988356096
   type: freebsd
   index: 2
   end: 3860639
   start: 1930383
3. Name: ada0s3
   Mediasize: 1548288 (1.5M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1976647680
   Mode: r0w0e0
   rawtype: 165
   length: 1548288
   offset: 1976647680
   type: freebsd
   index: 3
   end: 3863663
   start: 3860640
4. Name: ada0s4
   Mediasize: 21159936 (20M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1978195968
   Mode: r1w1e2
   rawtype: 165
   length: 21159936
   offset: 1978195968
   type: freebsd
   index: 4
   end: 3904991
   start: 3863664
Consumers:
1. Name: ada0
   Mediasize: 12884901888 (12G)
   Sectorsize: 512
   Mode: r2w1e4

Geom name: ada0s1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s1a
   Mediasize: 988283392 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 40448
   Mode: r1w0e1
   rawtype: 0
   length: 988283392
   offset: 8192
   type: !0
   index: 1
   end: 1930256
   start: 16
Consumers:
1. Name: ada0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r1w0e1

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: 8a6e5b83-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da0p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: 8a8d757f-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da0
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: 8b4a98ea-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da1p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: 8b67d5b2-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da1
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da2
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: 8c26911f-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da2p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: 8c45976e-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da2
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da3
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: 8d041090-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da3p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: 8d24b451-2ca9-11e4-865b-001b78595db4
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da3
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3

Geom name: da5
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da5p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: 2a5833d5-fa8d-11e4-bfd3-32a10598571e
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da5p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r1w1e2
   rawuuid: 2a6da0e6-fa8d-11e4-bfd3-32a10598571e
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da5
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r2w2e5

Geom name: da7
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da7p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: d1a05d34-fdbb-11e4-a38e-32a10598571e
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da7p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r1w1e2
   rawuuid: d1bb59ac-fdbb-11e4-a38e-32a10598571e
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da7
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r2w2e5

Geom name: da4
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da4p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: fd7f34f2-ff2e-11e4-a38e-32a10598571e
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da4p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r1w1e2
   rawuuid: fd930d38-ff2e-11e4-a38e-32a10598571e
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: da4
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Mode: r2w2e5

Code: 
 pool: zfs
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 4.34T in 9h18m with 5 errors on Thu May 21 15:39:15 2015
config:

        NAME                                              STATE     READ WRITE CKSUM
        zfs                                               ONLINE       0     0    35
          raidz2-0                                        ONLINE       0     0    70
            gptid/2a6da0e6-fa8d-11e4-bfd3-32a10598571e    ONLINE       0     0     0
            gptid/8a8d757f-2ca9-11e4-865b-001b78595db4    ONLINE       0     0     0
            gptid/8b67d5b2-2ca9-11e4-865b-001b78595db4    ONLINE       0     0     0
            gptid/8c45976e-2ca9-11e4-865b-001b78595db4    ONLINE       0     0     0
            replacing-4                                   ONLINE       0     0     0
              gptid/8d24b451-2ca9-11e4-865b-001b78595db4  ONLINE       0     0     0
              gptid/fd930d38-ff2e-11e4-a38e-32a10598571e  ONLINE       0     0     0
            gptid/d1bb59ac-fdbb-11e4-a38e-32a10598571e    ONLINE       0     0     0

errors: 5 data errors, use '-v' for a list
[root@freenas] ~#


it is freenas but due to some disagreements i am not really welcom on the freenas forum.
 
to add more confusion
148k6xl.png
[/IMG]

2dsrbf7.png
[/IMG]
 
You have real problems if you have errors at the vdev and zpool level which you do. Also you can see that you have permanent errors.

I would recreate the pool entirely.
 
would this have been caused by a crap drive/s, or by some operator error?
 
my ZFS experience? or is there some 'ZFS History' log that i am unfamiliar with?

i have only been using ZFS for around 1 year. i have known and seen and been reading about ZFS for maybe 5 years.

in my trials i have suxessfully replaced a few drives, and physically moved an array from a VM to a standalone server, and back to a VM. (just to test that i could complete the task succesfully) so i am no expert. but i havecompletly lots of the comon ZFS tasks before this array when kaboom.

i also tested several different ZFS installs and moved an array between them before landing on freenas.
 
see, still fairly ZFS n00bish.... heres the results.
i did do some playing with freenas jails but i really do not like them, i moved the intended purpose to dedicated headless VMs and saved a ton of resources. so all the jails should be gone.

Code: 
[root@freenas] ~# zpool history zfs
deleted history file to preserve path data from gleaning of useful (useless) personal info
[root@freenas] ~#
 
Last edited:
i realize this was probably caused by me but i would like to hear THEORY as to WHAT i flubbed up so that i can avoid it in the future, i am already in the process of restoring most of the data and scavenging some of the less concerning stuff so saving this array is not an issue. if there is things that i could try to further be a learning experience that would also be great.
 
i am going to update with some more info just as a closing to this issue.

for some more trouble shooting steps i tried shuffling the drives across different controllers, i have also done checked the ECC logs on the RAM, done scrubs on my other arrays on these controllers and the only thing that is consistent is the group of drives tied to this one array. also, it is a fairly tightly packed grouping of folders where the corruption always appears, and the corruption did not start appearing until recently. (space had been slowly growing) my GEUSTIMATION is a drive (drives) with bad sectors that the drive could not map around, so any time data ended up there and was called back ZFS would issue read errors. (smart check with the smartctl technically passes but i do not think it looks good on any of the drives in the old array)

i ended up with new drives, a new array, a partial backup restore (critical stuff) and salvaged some other stuff from the bad array that was less than critical. overall i might have lost 0.1% (i do have a large group of ISOs and random junk that is not critical so not backed up in any way) of any data and ZFS continually did its best to compensate.

HOPEFULLY that is the case and i am golden again.
 
Last edited:
You must log in or register to reply here.
Back
Top