• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS Drive Replace Issue

N

No1451

Weaksauce
Joined
Oct 19, 2010
Messages
75
One of my drives recently kicked the bucket, total and utter failure as far as I can tell. While in process of replacing it with the RMA drive, another drive from that same vdev has decided to fail.

It isn't a consistent failure, however once every few hours(not sure what dictates this) it will randomly drop from the array. When it does this it kicks the resilver back to 0 and the whole process restarts....so the new drive can never be fulled completely added to the array.

I'm looking for any possible solution that doesn't involve backing up and rebuilding from scratch, I don't have the drives on hand to do it and I can't justify spending $700 or more on the drives necessary.

Setup:

The drives are connected in a norco RPC-3116 on 2 Intel SASUC8I controllers in with HBA firmware on them.
 
What type of vdev? If it is still redundant, pull that 2nd drive out so the resilver can finish?
 
What about using another drive to replace that intermittent one? As in, pull the marginal drive and dd it's contents to a fresh one. Then put the new one in it's place. Don't know whether or not ZFS would allow this trickery or not.
 
I was thinking about that, but like you say I'm not sure how ZFS determines drive identities. If possible that is exactly what I will do. hopefully it is possible and someone who knows for sure can chime in.
 
How many drives do you have in the array? How much data do you have?

What PSU are you using? Do any of the drives have loose connections? Do you have good airflow in your case or are all the drives cooking?

First thing I would check is that the drives have adequate power to all of them. Next I would check all the data connections to see if vibrations dislodged them. Are they on a backplane? Is the backplane going out?

Unfortunately if a second drive is failing your options are becoming sparse. I can think of 2 options:

1) get some free space and copy everything over.
2) get an identical drive to the one that is now dropping in and out and dd copy everything over to it. Then switch the boards on the drives and hope it works.
 
4 drives in this vdev, the drive temperatures are all nominal and the PSU should be sufficient(850watt). I've checked for physical issues at the moment it certainly seems like its the drive, however the backplane could very well be the point of failure. I will connect them all direct to the controller and see if there is any change.

So I'm guessing since you say switch boards that ZFS uses the serial number or some other unique identifier on the drive to ID them? Copying over simply isn't an option, I don't have 10TB free anywhere in the house sadly.
 
No1451 said:
I was thinking about that, but like you say I'm not sure how ZFS determines drive identities. If possible that is exactly what I will do. hopefully it is possible and someone who knows for sure can chime in.
Click to expand...

It depends on if you are using a SAS HBA or not.

If you are using motherboard SATA ports then yes it is that simple and you can use any similar drive to simply copy all contents over.

If you are using a SAS HBA then your HBA makes a unique drive identifier based on the drive's serial number and firmware version. If this is the case then you will have to get an exact drive replica and switch the boards like I mentioned above.
 
SATA drives, one and all. Thank you so much, I will give this a try just as soon as I have the replacement drive in my hands. Any particular recommended tool to copy? I'm on windows but can use something else if there is a better option.

Edit: Should be noted I suppose that the pool has a boot pool contained on it, so the drive has two partitions to copy over, I'm kind of ignorant of Linux or any methods to copy this over so I have no idea if that poses a significant issue.
 
One of the easiest ways to deal with drives is with a linux live boot CD that has the gparted program on it. This way you're not booting into the host OS at all, and using an OS that won't disturb what's already on the drives. Assuming you don't screw up, that is. A very safe route would be putting the marginal drive and a new one into a whole other PC. Then use a linux boot CD. Depending on where the drives show up it could be as simple as 'dd if=/dev/sda of=/dev/sdb' where /dev/sda is your source drive and /dev/sdb is your new one. Using a bare-bones setup helps avoid mistakes; you have fewer devices showing up so there's less chance of mistakes.
 
Alright, sounds like a plan. Thanks for the info, I will be doing this just as soon as the RMA drive shows up
 
You must log in or register to reply here.
Back
Top