• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS pool unavailable

L

Lui

n00b
Joined
Aug 18, 2011
Messages
15
Hm sound serious this time. The server was down for two days. Today I've stared it and one of two pools is unavailable. Here is zpool status:

pool: tank
state: UNAVAIL
status: One or more devices are faulted in response to persistent errors. There are insufficient replicas for the pool to
continue functioning.
action: Destroy and re-create the pool from a backup source. Manually marking the device
repaired using 'zpool clear' may allow some data to be recovered.
scan: none requested
config:

NAME STATE READ WRITE CKSUM
tank UNAVAIL 0 0 0 insufficient replicas
raidz1-0 UNAVAIL 0 0 0 insufficient replicas
c2t5d0 UNAVAIL 0 0 0 cannot open
c2t0d0 UNAVAIL 0 0 0 cannot open
c2t1d0 FAULTED 0 0 0 too many errors
c2t4d0 UNAVAIL 0 0 0 cannot open

What is going on?
 
Not a whole lot to go on. Are you sure all the devices are available and good? Ditto controllers, etc...
 
Well I'm not sure. Those 4 HDD are on motherboard SATA controller and drives are detected in BIOS. They are also listed with format command. I really don't understand why they have cannot open status. I'll try to switch them on AOC SAS2008 controller tomorrow. BTW I'm using Openindiana 151a and 4x2TB Hitachi 5k3000 HDDs in this pool.
 
Last edited:
Yes I've tried with zpool clear but since zpool has unavail status nothing can be done.

fmadm faulty listing:

Code: 
--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Mar 08 20:15:03 c18d697b-348e-6bc7-c9bf-c208f27603bc  ZFS-8000-CS    Major     

Host        : storage
Platform    : System-Product-Name	Chassis_id  : System-Serial-Number
Product_sn  : 

Fault class : fault.fs.zfs.pool
Affects     : zfs://pool=tank
                  faulted and taken out of service
Problem in  : zfs://pool=tank
                  faulted and taken out of service

Description : A ZFS pool failed to open.  Refer to
              http://sun.com/msg/ZFS-8000-CS for more information.

Response    : No automated response will occur.

Impact      : The pool data is unavailable

Action      : Run 'zpool status -x' and attach any missing devices, follow
              any provided recovery instructions or restore from backup.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Mar 08 18:30:52 a407f7c4-8533-e03c-91e6-8b70a943a886  ZFS-8000-D3    Major     

Host        : storage
Platform    : System-Product-Name	Chassis_id  : System-Serial-Number
Product_sn  : 

Fault class : fault.fs.zfs.device
Affects     : zfs://pool=tank/vdev=df8000dd7a835982
                  faulted and taken out of service
Problem in  : zfs://pool=tank/vdev=df8000dd7a835982
                  faulted and taken out of service

Description : A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for
              more information.

Response    : No automated response will occur.

Impact      : Fault tolerance of the pool may be compromised.

Action      : Run 'zpool status -x' and replace the bad device.
Hm they (Sun/Oracle/Solaris guys) are suggesting an export/import zpool procedure. I've never done that. Should I go for it?
 
Last edited:
zpool clear tank returns:

Code: 
:~# zpool clear tank
cannot clear errors for tank: no such device in pool
 
Export and import should be harmless, give it a try. Also, can you clarify something? You said the server was down for a couple of days. Crash? Power off? Shutdown? Hardware failure?
 
Just shutdown, no drama. But guess what happened after import/export?

This: :)))))))

Code: 
:~# zpool list
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
medias  3.62T  1.07T  2.55T    29%  1.00x  ONLINE  -
rpool   55.5G  5.52G  50.0G     9%  1.00x  ONLINE  -
tank    7.25T  5.13T  2.12T    70%  1.00x  ONLINE  -
lojze@storage:~# zpool status -v tank
  pool: tank
 state: ONLINE
  scan: scrub repaired 0 in 6h31m with 0 errors on Sun Mar  4 05:31:47 2012
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    c13d1   ONLINE       0     0     0
	    c11d0   ONLINE       0     0     0
	    c12d0   ONLINE       0     0     0
	    c13d0   ONLINE       0     0     0

errors: No known data errors

I think that backup is now the next procedure and then scrub, right?
 
No, I would scrub first. If the scrub is going to fail, your backup is likely crap. Good to hear you got the pool back.
 
And thank you for your support. The fmadm faulty was an eyeopener. I'll sleep really well now Good night! (Europe - Slovenija = it is dark)
 
You must log in or register to reply here.
Back
Top