ZFS pool SUSPENDED can it ben save

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
pool: tank
state: SUSPENDED
status: One or more devices are unavailable in response to IO failures.
The pool is suspended.
action: Make sure the affected devices are connected, then run 'zpool clear' or
'fmadm repaired'.
Run 'zpool status -v' to see device specific details.
see: http://support.oracle.com/msg/ZFS-8000-HC
scan: resilvered 221M in 0h16m with 0 errors on Mon May 4 10:12:55 2015
config:

NAME STATE READ WRITE CKSUM CAP Product
tank SUSPENDED 0 0 0
raidz2-0 UNAVAIL 0 0 0
c0t50014EE257E90175d0 UNAVAIL 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE2058EA559d0 UNAVAIL 0 0 0 2 TB WDC WD20EARS-00M
c0t50014EE202933839d0 ONLINE 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE2AD3EA42Ed0 UNAVAIL 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE257E8B376d0 ONLINE 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE25AE3B67Ed0 ONLINE 0 0 0 2 TB WDC WD20EARS-00M
raidz2-1 UNAVAIL 0 0 0
c0t50014EE20A713053d0 UNAVAIL 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE20A711435d0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE25FC6912Ed0 UNAVAIL 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE2B51C2C56d0 UNAVAIL 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE20A714600d0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE25F5FA45Ad0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W

Hi I got this from my zpool cabels look ok the 4 TB hdd have not run that much
I haqve google but I will rely dont fu...k it up so sorry if I ask stupet
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,066
Half of your disks are unavailable.
I would power off and check all cables and power connectors.

When disks come back, you need to do a zpool clear tank and the pool is back online.
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
I have check all cabels one more time now
now
ool: tank
state: UNAVAIL
status: One or more devices are unavailable in response to persistent errors.
There are insufficient replicas for the pool to continue functioning.
action: Destroy and re-create the pool from a backup source. Manually marking
the device repaired using 'zpool clear' or 'fmadm repaired' may
allow some data to be
recovered.
Run 'zpool status -v' to see device specific details.
scan: none requested
config:

NAME STATE READ WRITE CKSUM CAP Product
tank UNAVAIL 0 0 0
raidz2-0 UNAVAIL 0 0 0
c0t50014EE257E90175d0 UNAVAIL 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE2058EA559d0 UNAVAIL 0 0 0 2 TB WDC WD20EARS-00M
c0t50014EE202933839d0 ONLINE 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE2AD3EA42Ed0 UNAVAIL 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE257E8B376d0 ONLINE 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE25AE3B67Ed0 ONLINE 0 0 0 2 TB WDC WD20EARS-00M
raidz2-1 DEGRADED 0 0 0
c0t50014EE20A713053d0 UNAVAIL 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE20A711435d0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE25FC6912Ed0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE2B51C2C56d0 UNAVAIL 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE20A714600d0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE25F5FA45Ad0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W

so it loook like it lost but hay it only data and got backup :)
 

drescherjm

[H]F Junkie
Joined
Nov 19, 2008
Messages
14,935
so it loook like it lost but hay it only data and got backup :)

I do not think it is lost at all. Unless 6 of your drives were all destroyed at the same time. I would call that highly unlikely unless your power supply supplied the wrong voltage for these 6 but did not for the rest. Wrong modular power cable used?. This seems like a cabling issue.
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
I do not think it is lost at all. Unless 6 of your drives were all destroyed at the same time. I would call that highly unlikely unless your power supply supplied the wrong voltage for these 6 but did not for the rest. Wrong modular power cable used?. This seems like a cabling issue.

also whit the news status ? the state op the disk UNAVAIL are on some new one can get why I have chik the backpalate cabel and it look good in place :-(
 

drescherjm

[H]F Junkie
Joined
Nov 19, 2008
Messages
14,935
also whit the news status ? the state op the disk UNAVAIL are on some new one can get why I have chik the backpalate cabel and it look good in place :-(

Try to determine what is common between the 6 drives that are UNAVAIL. Perhaps the backplane is defective.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,066
Replace one of the unavail disk with a working one and check if its working there.
Unless you have an overvoltage problem that kills 6 of 12 disks, you have a cabling, power or controller problem.

If you look at your second status screen your 6 unavail disks are reduced to 5 unavail.
I doubt that the disks are the problem.
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
Replace one of the unavail disk with a working one and check if its working there.
Unless you have an overvoltage problem that kills 6 of 12 disks, you have a cabling, power or controller problem.

If you look at your second status screen your 6 unavail disks are reduced to 5 unavail.
I doubt that the disks are the problem.
Ok I have try to change the plase of my HDD so the backplate can not be it , then I try to take a HDD I know work and swits one of the 2TB hdd. but dont know or it can se it.
cant run zpool clear that devices is currently unavailable im realy lost
when I run fmadm faulty and run fmadm repaired for the FRU I get the status to repired but cant get it to be save it stell just markt Status : repaired when run fmadm faulty and all of the disk that are UNAVAIL stell are sorry im are a noob I realy try to read and read up on it


pool: tank
state: UNAVAIL
status: One or more devices are unavailable in response to persistent errors.
There are insufficient replicas for the pool to continue functioning.
action: Destroy and re-create the pool from a backup source. Manually marking
the device repaired using 'zpool clear' or 'fmadm repaired' may
allow some data to be
recovered.
Run 'zpool status -v' to see device specific details.
scan: none requested
config:

NAME STATE READ WRITE CKSUM CAP Product
tank UNAVAIL 0 0 0
raidz2-0 UNAVAIL 0 0 0
c0t50014EE257E90175d0 UNAVAIL 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE2058EA559d0 UNAVAIL 0 0 0
c0t50014EE202933839d0 ONLINE 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE2AD3EA42Ed0 UNAVAIL 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE257E8B376d0 ONLINE 0 0 0 2 TB WDC WD20EADS-00R
c0t50014EE25AE3B67Ed0 ONLINE 0 0 0 2 TB WDC WD20EARS-00M
raidz2-1 DEGRADED 0 0 0
c0t50014EE20A713053d0 UNAVAIL 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE20A711435d0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE25FC6912Ed0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE2B51C2C56d0 UNAVAIL 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE20A714600d0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
c0t50014EE25F5FA45Ad0 ONLINE 0 0 0 4 TB WDC WD40EFRX-68W
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,066
If a disk is unavail, its missing.
You cannot "clear" such an error.

- Power off your system and unplug all disks
- Use a single functional disk port from your HBA and check all 12 disks there

Your HBA seems hot plug capable, you do not need to power down on a switch.
If you hot-plug/unplug a disk. wait some seconds and check if its then visible ex for a format command.
If it becomes visible, cancel the format command via ctrl-c and check for next disk

You need to know if your disks are damaged or ok, then you can check for HBA, power or other problems.
 

Master_shake_

Fully [H]
Joined
Apr 9, 2012
Messages
17,795
did anyone else notice that 6 of the drives were down.

but then in the second one only 5 were down?
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
If a disk is unavail, its missing.
You cannot "clear" such an error.

- Power off your system and unplug all disks
- Use a single functional disk port from your HBA and check all 12 disks there

Your HBA seems hot plug capable, you do not need to power down on a switch.
If you hot-plug/unplug a disk. wait some seconds and check if its then visible ex for a format command.
If it becomes visible, cancel the format command via ctrl-c and check for next disk

You need to know if your disks are damaged or ok, then you can check for HBA, power or other problems.

ok it weird I do as you said, nice to have hot plug when you do that :D but I got 5 hdd that get UNAVAIL no matter what port I put them in and know the port work for the order work in that port so what can have happen whit that 5 hdd I can se all hdd format but that 5 are UNAVAIL no matter what I do
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,066
do not quite understand what you mean by that but I can se the disk whit format and also ind my zfs pool whit S/N but there are also show like UNAVAIL

Thats essential.

If you can see all single disks with format, then they are working.
If they are missing/unavail when you connect them all, you have a power, hba or backplane problem.

You can try to connect the unavail disks directly to sata/AHCI.
If they are working there, your backplane, or HBA is the problem,
otherwise replace your power supply.
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
Thats essential.

If you can see all single disks with format, then they are working.
If they are missing/unavail when you connect them all, you have a power, hba or backplane problem.

You can try to connect the unavail disks directly to sata/AHCI.
If they are working there, your backplane, or HBA is the problem,
otherwise replace your power supply.

ok thx but how can it be a hba or backplane or powersupply problem if the are the same if I put the unavail disk ind the same plase where a disk is not unavail so are the only thing there hace change the disk all the orther are the same :( I really appreciate your trying to help me
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,066
If single disks are detected properly but some are missing when using them all together,

- you have not enough power
- you have a cabling, backplane or HBA problem
- one disk is blocking the bus hindering the others to be detected ot to stay available.

You can also try the following
insert disk by disk and see if the new one is detected and the others keep available
and/or use Sata for the missing disks
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
If single disks are detected properly but some are missing when using them all together,

- you have not enough power
- you have a cabling, backplane or HBA problem
- one disk is blocking the bus hindering the others to be detected ot to stay available.

You can also try the following
insert disk by disk and see if the new one is detected and the others keep available
and/or use Sata for the missing disks

"- you have not enough power the PSU are a 1000 watt and guld standart so dont think it can be that
" - you have a cabling, backplane or HBA problem" when a woking disk place are swits whit a disk that are UNAVAIL it are stell UNAVAIL

"one disk is blocking the bus hindering the others to be detected ot to stay available." look up

only thing I can try are sata only but dont get why it will help and it vill be hard when you got the backplate :-( but thx for all you help :)
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
Fire_Shot_Capture_solaris_ZFS_appliance_h.png


ok I try to put it on sata and power cabel the hdd whit id nummer c4t16d0 all down on the site are the same as c0t50014EE20A713053d0 so it must be the cabel but can get why when it not help to change the plase
 
Last edited:

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,066
ok I try to put it on sata and power cabel the hdd whit id nummer c4t16d0 all down on the site are the same as c0t50014EE20A713053d0 so it must be the cabel but can get why when it not help to change the plase

On Solaris, disks on a professional LSI HBA in IT mode are shown via their WWN number c0t50014EE20A713053d0 .
This number is disk unique and keeps the same when you move such a disk around (And keeps known as a raid member).

In contrast to this, you have port numbers like c4t16d0 what means Controller 4, port 16.
If you move such a disk to another port ex another Sata port, it is detected under the new number.

In such a case, the raid tells you that the old id is unavail and the new number not part or the raid.
To fix this, you need to export/import the pool as in this case all disks are re-read and the ZFS config is created newly.
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
On Solaris, disks on a professional LSI HBA in IT mode are shown via their WWN number c0t50014EE20A713053d0 .
This number is disk unique and keeps the same when you move such a disk around (And keeps known as a raid member).

In contrast to this, you have port numbers like c4t16d0 what means Controller 4, port 16.
If you move such a disk to another port ex another Sata port, it is detected under the new number.

In such a case, the raid tells you that the old id is unavail and the new number not part or the raid.
To fix this, you need to export/import the pool as in this case all disks are re-read and the ZFS config is created newly.
ok thx so the disk work. but not work if it use the backplate even if the backplate have the same disk number and place weird but nice hearing was so close to say f...k it and go whit SnapRAID and take a litel data lost and take a backup
 

tobiasl

Weaksauce
Joined
Nov 19, 2007
Messages
100
ok I throw the towel and get my backup thx for all you help and trining to help me I relay happy for it
 

drescherjm

[H]F Junkie
Joined
Nov 19, 2008
Messages
14,935
ok I throw the towel and get my backup thx for all you help and trining to help me I relay happy for it

I expect the restore to fail since you have a hardware problem preventing 1/2 of your drives from being seen by the OS (when all are connected).


Edit:
Or did the drives get reassigned new ids?
 
Last edited:

Shockey

2[H]4U
Joined
Nov 24, 2008
Messages
2,240
my case is a X-case but the Backplane look like http://www.norcotek.com/item_detail.php?categoryid=1&modelno=rpc-4224 can se it have 2 molex for every backplane so maby it that ? it have use only one but steel dont get why it cant work whit only one drive so

the X-case are similar to Norco. I know Norco uses the second molex for redundancy. I'd check your manual before throwing in towel, sounds like you have hardware problem ;) Backup isn't going to save you here...
 
Top