Force-Add Drive To Areca Array

[Hardy]

We have an Areca 1680ix-24 with an Areca SAS Expander card. There is a 24-drive RAID 6 array using all 24 Expander card slots. At least once a week, the controller card will sense one of those drives being "removed" then immediately "reinserted", causing an array rebuild. {Would love to hear how to prevent that from happening.}

During the latest rebuild, the server had to be hard-powered off and restarted. Upon restart a different drive from the same array was showing as removed from the array (it was "free"), resulting in an "Incomplete" array with 23 of 24 member disks.

Is there a way to force-add this second drive to the array in order to reactivate and then rebuild the array?

Physically removing and reinserting the drive does not do it -- the drive remains "free".

I can set the drive as a hot spare, but then it will have to rebuild 2 drives (doubling the time it takes to rebuild the array, risking a third drive being "removed"/"reinserted" and breaking the array completely). Plus, I don't know that the "Incomplete" array will automatically pick up a hot spare.

Thanks for your suggestions.

.

 

[mwroobel]

First, you would never want to force add a drive to an array, because you can't be sure of the quality of the data on the dropped drive. The whole purpose of RAID is to allow you uptime while the array is degraded, and to rebuild from known good data that exists in the array as it stands.
You should always have a full backup of your data... RAID IS NOT A BACKUP STRATEGY! As to the drive(s) being dropped, are they always the same slot, or are they all coming from the same cable tree or same backplane? What expander model are you using, and what enclosure are these drives in?

 

[Hardy]

Thank you for the reply.

I agree with your primary points -- RAID is not a replacement for backups, and generally you don't want to just add a dropped drive back into an array.

The Expander card is an Areca 8026-24, and most removal/reinserts are to drives on the same cable (I've switched it out) and backplane, but not always.

To eliminate heat as a contributing factor, we relocated the drives from a 24-bay Norco case to open shelving with an external cooling fan.

As to the issue with the array, the array broke when a drive was dropped at the point of power off (or BIOS power on), so we are reasonably certain the drive's data is intact. Certainly if we could add the drive to the array we could check it for parity errors.

However, neither unplugging then replugging the drive nor setting the drive as a hot spare causes the array to re-acquire that drive. The result is that the array remains in an "Incomplete" state, with only 23 of 24 members.

Since the 1680 is not automatically adding the "free" drive to the array, is there a way to force the controller to do so?

Or, what other steps might we take (short of a start-from-scratch restore from backup) to restore the array?

Thanks for helping with this.

.

 

[odditory]

Unless there's an undocumented areca rescue-keyword or CLI command I'm unaware of then there's no way to force-add a drive back to an array, short of deleting and recreating the volume in NO-INIT mode -- which is dangerous and only applies to very specific last-resort type scenarios since user error will tend to happen and mistakes can easily be made by not doing everything exactly right (replicating original drive order, original stripe size, original raid level, etc) can make a bad situation worse.

First of all don't sweat it you're not going to lose any data if you take it one step at a time. Where people run into trouble in these situations is when they panic and start experimenting or trying things without an understanding of the ramifications and risks of each particular step. Now you haven't stated which harddisks you're using. There happen to be issues with certain make/models of harddisks and 1680 series cards or at least there used to be, whereby they'd randomly drop - specific Seagates and WD's especially. Also, what all is connected to the Areca - just the expander, or expander + other drives on the Areca's onboard ports? Please detail your cabling topology a little more.

Also, its unclear how many drives are missing from the array - you say 23/24 are present but OTOH say you dont want a double rebuild. Could you post a screenshot of your Information->Raidset Hierarchy page from the web gui? The drives sitting on an "open rack with external cooling" sounds a little scary, btw. Also, are these drives holding business/enterprise data or just home oriented data like movies? I ask because I'd consider a 1680 legacy and EOL for business use and you'd want to upgrade to an 1880 or 1882 without wasting any more time. However for home use and things like video streaming a 1680 is still fine.

You'd also do well to upgrade the card to firmware 1.51 which came out recently if you have not already - certain behaviors and idiosynracies of previous revs have been tweaked/improved as well as there is a new drop-down option for defining what constitutes a hot spare (any free drive/no drive/only free drives that are blank).

Lastly, under normal circumstances, if a drive has dropped and shows back up as free and you want to add it back to the array all you'd do is set it as a hotspare dedicated to the raidset in question. The rebuild would start after that. But if it were me then the first priority would be determining the hardware is all stable before beginning a rebuild.

 

[Hardy]

Thanks for the reply.

Yes, we had to do the no-init before and it was tedious. Kevin Wang was critical to our success. We don't want to repeat that experience.

The 1680ix-24 has a 6-disk boot array (RAID 6) and a 16-disk data array with hot spare (RAID 6), in addition to the SAS Expander with 24-disk RAID 6 array (containing non-critical though large quantity and difficult to replace business data).

Originally we had mostly Seagate and WD green drives, and as those dropped we replaced many of them with Seagate black. Now even the Seagate blacks drop.

We are not fans of the open rack situation either, but when we had all 24 drives in the Norco case we would have constant and multiple dropouts -- not sure if it was due to vibration or heat or orientation (the case was vertical) or something else, but at least now we get dropouts only every week or two.

We can upgrade the 1680 firmware, but to your point that we should take this one step at a time, is it recommended that we upgrade before resolving the immediate array issue?

Considering our only trouble with the 1680 is the drives dropping out of the array, do you think the 1880-series would be better in that regard?

To get to this point, the controller sensed 1 drive "removed" then "reinserted" and began an array rebuild -- this takes 2-3 days. It was during the rebuild that the hard-power-down caused the second drive to be dropped from the array (making it "Incomplete"). Thus, if the second drive is added back in as a hot spare, the first drive must finish being rebuilt before that second (hot spare) drive is then rebuilt -- double rebuild. Given the long rebuild times, this risks a third simultaneous drive problem. If we can have the controller assume the dropped drive has good data, the array would not need to be rebuilt a second time.

We did try to set the dropped drive as a hot spare, but the controller did not automatically pick it up for the Incomplete array -- it remained "free". Do you know how long we have to wait for the array to add that drive to the array, or if there is a way to force-add the hot spare to the array?

Your advice and assistance is appreciated.


.

 

[mwroobel]

Seagate doesn't make a Black drive, maybe you are thinking of the WD black? The WD greens are unsuitable for HW RAID and the blacks as well (though not as problematic) because of how they handle (and how long they try) errors. Since this is a business server, do you have a complete backup of all three arrays? To be safest, I would recommend drives that are HW RAID friendlier (going with Enterprise drives from Seagate or Hitachi) or if you want to stay with consumer drives either the WD RED, hitachi 5k or 7k or the Seagate ST3000DM001.

 

[Hardy]

You are correct: WD black.

Our critical data is on enterprise drives, and we are working on a migration plan for the non-critical data.

Thanks for your advice.