Areca raid reliability

Joined
Sep 14, 2008
Messages
1,622
For quite a while I have praised areca controllers for their reliability compared to other brands. Unfortunately recently some things have happened which has caused me to have a change of heart...

We have had major issues with raid controller failures with both LSI and 3ware which is what prompted us to switch to areca. For the longest time we had no failures and just recently (2 months or so ago) various machines that have been running with no issues for 6 months or sometimes 1+ year are crapping out.

In linux the cards get arcmsr0: abort device command of scsi id = 0 lun = 1 and eventually will go to 'rejecting I/O to offline device'

I have seen this in the past caused by a bad drive getting timeouts or read errors and it was a matter of replacing the bad disk but recently its happened and there is nothing in the event log:

Code:
Date-Time            Device           Event Type            Elapsed Time Errors
===============================================================================
2011-02-16 22:36:42  H/W MONITOR      Raid Powered On
2011-02-16 12:31:49  H/W MONITOR      Raid Powered On
2011-02-16 11:09:53  H/W MONITOR      Power On With Battery Backup
2011-02-16 07:45:28  H/W MONITOR      Raid Powered On
2011-02-15 14:41:47  H/W MONITOR      Raid Powered On
2010-12-18 22:16:54  H/W MONITOR      Raid Powered On
2010-12-18 22:07:45  H/W MONITOR      Power On With Battery Backup
2010-12-18 21:43:01  H/W MONITOR      Raid Powered On
2010-12-18 11:14:52  H/W MONITOR      Raid Powered On

When a disk timeout occurred sometimes one of the disks would show a solid activity light but now a lot of times its like the controller is just crashing. Viewing the machine will show multiple random disks with hung disk activity lights. If you have a ethernet cable hooked up to it as soon as it crashes the web-interface will no longer be accessible. Here is a video I took of a server with a bad card and you can see the behavior:

http://www.youtube.com/watch?v=WNyjVWeehrk

This is right before I replaced the controller with a new one. So far replacing the controller with a new one has always fixed the issue suggesting its a problem with the hardware.

Most of the other brands of controllers we have had have failure rates between 15-25% (I am not joking) and until recently areca was <1% but now its getting up to about 5-7%. Its still much lower than the other brands but we have had problems with RMAing the cards. Areca checks out the cards and tell us there is nothing wrong with them =(. They said to run the newest driver and firmware and we were using the newest firmware but not driver. Even with the newest driver the problem persists and everything points to hardware issues.

We will likely be attempting to RMA again and hopefully get some new cards as it costs around $500 for each replacement card that we don't get an RMA for.

Anyway I just wanted to let people know my experience of the different brands as we buy in bulk. I have mentioned in the past areca has been much more robust/reliable than the other brands and now it appears they are more reliable but not 'much more' and they don't seem to be very good with RMA's even with a company that has over 500 of their controllers

Here is another vid I took of a bunch of dead raid controllers. We only ever had about 100 servers with 3ware (300 with LSI and 500+ areca) and you see we have a whole box of them and likely many others laying around.

http://www.youtube.com/watch?v=w8E2cu26jsI
 
Very interesting thanks for writing all of this out.

I've had the arcmsr0: abort device error as well but only when a drive was acting up and didn't get kicked from the array.
 
Very interesting thanks for writing all of this out.

I've had the arcmsr0: abort device error as well but only when a drive was acting up and didn't get kicked from the array.

Yeah Until a month or two ago that was the *only* time we saw that error. Simply pull the drive and the machine was good. Now the controller appears to just be failing which is very bad =(.

Btw xeonMP I remember helping you on your box for getting higher speeds out of your ARC-1680ix-24. I remember your opteron core wasn't fast enough for DD to get more than 500 MB/sec or so. If you bypass the disk cache and use dd with iflag=direct then it uses *way* less CPU usage. I found out a newer intel model was limited by CPU usage with DD at around 1GB/sec and with direct I/O it could get 1.5 gigabytes/sec (this was an ARC-1880i).

I hope your server has been working good for you =)
 
Your controller has too strict timeout so the disks get detached on an active bad sector (Current Pending Sector in the SMART output for that drive).

I have the same thing with my Areca with firmware 1.42 or so. Didn't test with the most recent firmware, and there's no setting to control the timeout.

So you can only use Areca with TLER-enabled drives which is a real bummer!
 
Your controller has too strict timeout so the disks get detached on an active bad sector (Current Pending Sector in the SMART output for that drive).

I have the same thing with my Areca with firmware 1.42 or so. Didn't test with the most recent firmware, and there's no setting to control the timeout.

So you can only use Areca with TLER-enabled drives which is a real bummer!

I have seen when that is the issue and the behavior is completely different. This is definitely NOT what is happening. If you re-read my post you will see that replacing the controller fixes the issue.

I have seen a machine where within 20 minutes of booting its having issues and its powercycled, and again <20 min having issues again... And after running into that 2 or 3 times I replace it with a new controller and bam.. problem gone for good. Also I did not see times where 5/8 drives LEDs were solid when it was caused by a bad disk. Also when it was a bad disk the web management could still be accessed via OOB management over ethernet and the controller could be pinged (in this case it can't be)
 
So what is actually wrong with the cards, don't you just have some cable issues? Did you check SMART output for "UDMA CRC Error Count" or other errors?

I mean, if Areca tells you the cards you send RMA are working fine, then i wonder what is going on? Your videos didn't tell me much as to what really happens.
 
@houkouonchi: I wouldn't just be writing on a forum about it, i'd be taking it up with Kevin in tech support and their CEO, I can give you his direct email. I have had a similar experience sending something in to Tekram which is Areca's U.S. based distro and tech support, only to have them bounce it back with "we couldn't find anything wrong." In my case it was a bad port on one of the onboard miniSAS connectors, but the problem would only show up if using a SFF-8087 to SFF-8087 cable. If using an SFF-8087 to SATA breakout, there was no problem and because they tested with a breakout cable they couldn't find the problem. And this after I spent a long time eliminating bad cables or backplanes as possible culprits. The problem followed the card.

As for your issues I have seen instances where a controller will just hard lock and require a reboot on certain false-positive drive events, which sounds like what you're describing. So at the least Areca needs to become aware of it regardless of the fact that Tekram plugs some drugs in and everything looks fine to them. That and the fact you're an enterprise customer with many of their cards.
 
Last edited:
Well we are in talks with them again now that we have tested witht he newest driver so I am hoping something will happen here.

In my case the only thing I am switching is the card and the cables stay. All of our areca machines are also using SFF-8087 breakout cables.

I definitely wasn't writing the forum for a fix. I just have been a been proponent of areca especially in the field of reliability on this forum and that recently changed so I wanted to let people know that.
 
I definitely wasn't writing the forum for a fix. I just have been a been proponent of areca especially in the field of reliability on this forum and that recently changed so I wanted to let people know that.

I understood that, and I appreciate your heads-up. Please keep us informed if you find out anything interesting.
 
@houkouonchi

I'm curious. You now seem to have had problems with all of the major brands of RAID card (Areca, 3Ware, Adaptec, LSI). I'm wondering if these guys all have their cards manufactured at the same plant (Foxconn?). If you decide to change brands, who's left?

Also, do the high-end SANs suffer these problems? Systems like Equallogic, EMC and Hitachi Data Systems; what controllers do they use and who makes them?
 
Back
Top