ARC-1880i on Linux/Open Solaris

H

houkouonchi

RIP
Joined
Sep 14, 2008
Messages
1,622
We have about 450 areca controllers where I work and I wanted to get the jump on testing the new ARC-1880i as we might switch to it if its compatible with the various drives we use and if it comes down in price a little bit.

My firsts test will be testing 1.5 TB drives using the SAS expander built into the SuperMicro 24 disk bay, First on linux and then on opensolaris. I will later be testing various drives types hooked up directly to the controller.

I am starting by using these 1.5 TB drives in my ARC-1231ML without any issues but most of our machines have ARC-1222's which I have not tested them on yet. I know these drives work on open solaris with this SAS expander on LSI controllers so it will be interesting to see how they work. I will also be testing 2TB hitachi's and 1TB seagate (333AS, 528AS, 340AS). The only disks that the ARC-1222 had issues with were the 340AS.

I will first Run a 24 disk array in raid0 off one SFF-8087 port just to see how that bottlenecks, I then will then setup a raid60 array of two sets of raid6 volumes (24 drives each) to see what kind of sequential read/writes I can get. I have a feeling I am gonna max out a core on DD though. I will let people know what the compatibility is of all the disks I test though.

arc1880i_0.png


arc1880i_1.png


More updates to come.
 
Linux Benchmarks so far:

Setup a 60 Tib (66TB) raid60 volume which took a little over 30 hours to init (long):

Code: 
2010-10-10 10:12:37  H/W MONITOR      Raid Powered On
2010-10-10 09:44:32  VOL#003R60Vol2-1 Complete Init         029:28:15
2010-10-10 09:12:45  VOL#003R60Vol2-2 Complete Init         028:56:29
2010-10-09 04:16:16  VOL#003R60Vol2-2 Start Initialize
2010-10-09 04:16:16  VOL#003R60Vol2-1 Start Initialize
2010-10-09 04:11:07  010.003.068.087  HTTP Log In
2010-10-09 04:09:25  E2 Fan 02        Failed
2010-10-09 04:09:25  E2 Fan 01        Failed
2010-10-09 04:09:25  E2 Fan 00        Failed
2010-10-09 04:09:25  H/W MONITOR      Raid Powered On
2010-10-09 04:08:10  VOL#003R60Vol2-2 Stop Initialization   001:48:04
2010-10-09 04:08:10  VOL#003R60Vol2-1 Stop Initialization   001:48:01
2010-10-09 02:20:09  VOL#003R60Vol2-1 Start Initialize
2010-10-09 02:20:08  VOL#000R60Vol2-1 Complete Init         000:02:13
2010-10-09 02:20:06  VOL#003R60Vol2-2 Start Initialize
2010-10-09 02:20:05  VOL#000R60Vol2-2 Complete Init         000:02:10
2010-10-09 02:18:07  ARC-1880-VOL#003 Create Volume
2010-10-09 02:17:55  VOL#000R60Vol2-2 Start Initialize
2010-10-09 02:17:54  VOL#000R60Vol2-1 Start Initialize

I have a feeling the SAS expander made this take longer than it normally would have...

My initial sequential read/write tests were definitely lower than I was expecting (especially in write speed).

Reads + 10 sec iostat output):

Code: 
livecd ~ # nice -n -20 dd bs=1M count=50000 if=/dev/sda of=/dev/null
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 43.4948 s, 1.2 GB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00   11.39    4.14    0.00   84.43

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
scd0             46.60     0.00    2.30    0.00     0.10     0.00    84.87     0.17   70.52  72.83  16.75
sda           336525.20     0.00 3749.80    0.00  1329.20     0.00   725.96     3.43    0.92   0.27  99.96

Writes:
Code: 
livecd ~ # nice -n -20 dd bs=1M count=50000 if=/dev/zero of=/dev/sda
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 82.0201 s, 639 MB/s


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.03    0.00   10.29   42.19    0.00   47.49

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda               0.00 191653.40    0.00 2104.80     0.00   761.04   740.51   104.70   49.90   0.45  94.43

I did notice that DD was using near 100% cpu usage during the tests though. I then later tried direct I/O which brought these values up quite a bit and DD used 20-30% cpu vs what it was using before:

Reads:
Code: 
livecd ~ # nice -n -20 dd iflag=direct bs=1M count=50000 if=/dev/sda of=/dev/null
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 38.2311 s, 1.4 GB/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.05    0.00    3.43    9.10    0.00   87.42

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
scd0              0.10     0.00    0.10    0.00     0.00     0.00     8.00     0.02  152.00 152.00   1.52
sda               0.00     0.00 2655.30    0.00  1327.65     0.00  1024.00     1.27    0.48   0.33  87.27

Writes:
Code: 
livecd ~ # nice -n -20 dd oflag=direct bs=1M count=50000 if=/dev/zero of=/dev/sda
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 63.796 s, 822 MB/s


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.05    0.00    4.54    7.99    0.00   87.42

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
scd0              0.10     0.00    0.10    0.00     0.00     0.00     8.00     0.02  223.00 223.00   2.23
sda               0.00     0.00    0.00 1566.00     0.00   783.00  1024.00     1.18    0.76   0.47  73.10


This is much better but to be honest the write speed is only marginally better than what my ARC-1280ML can get. It might be due to the fact its raid60 and having to do double parity calculations. I will be testing raid0 later and individual raid6 as well.

My file-system tests showed about just as good results as raw. Interestingly enough non direct reads didn't use nearly as much cpu usage (only about 50% so I didn't have to do direct I/O. I used JFS as the file-system as its pretty much just JFS and XFS that supports file-systems this large:

Reads:
Code: 
livecd jfs # nice -n -20 dd bs=1M if=./50gb.bin of=/dev/null
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 37.6557 s, 1.4 GB/s

Writes:
Code: 
livecd jfs # nice -n -20 dd oflag=direct bs=1M count=50000 if=/dev/zero of=./50gb.bin
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 66.8951 s, 784 MB/s

I also tested random reads/sec. I was also semi-disappointed about this as well but the SAS expander could be adding latency. For a benchmark my 20 disk system gets:

Code: 
root@dekabutsu: 11:27 AM :~# ./seeker_baryluk  /dev/sdd 128
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdd [69749987328 blocks, 35711993511936 bytes, 33259 GB, 34057611 MB, 35711 GiB, 35711993 MiB]
[512 logical sector size, 512 physical sector size]
[128 threads]
Wait 30 seconds..............................
Results: 1969 seeks/second, 0.508 ms random access time (1910701821 < offsets < 35711010979792)

This 48 disk system only got about 50% better results:

Code: 
livecd ~ # ./seeker_baryluk  /dev/sdb
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[1 threads]
Wait 30 seconds..............................
Results: 61 seeks/second, 16.295 ms random access time (9421241124 < offsets < 65915683335692)

livecd ~ # ./seeker_baryluk  /dev/sdb 2
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[2 threads]
Wait 30 seconds..............................
Results: 127 seeks/second, 7.853 ms random access time (22712282894 < offsets < 65901510049239)

livecd ~ # ./seeker_baryluk  /dev/sdb 4
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[4 threads]
Wait 30 seconds..............................
Results: 249 seeks/second, 4.014 ms random access time (6517916029 < offsets < 65916315711834)

livecd ~ # ./seeker_baryluk  /dev/sdb 8
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[8 threads]
Wait 30 seconds..............................
Results: 490 seeks/second, 2.039 ms random access time (8943144992 < offsets < 65910789319602)

livecd ~ # ./seeker_baryluk  /dev/sdb 16
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[16 threads]
Wait 30 seconds..............................
Results: 898 seeks/second, 1.113 ms random access time (3319844286 < offsets < 65918152951621)

livecd ~ # ./seeker_baryluk  /dev/sdb 32
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[32 threads]
Wait 30 seconds..............................
Results: 1541 seeks/second, 0.649 ms random access time (718141830 < offsets < 65919623338961)

livecd ~ # ./seeker_baryluk  /dev/sdb 64
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[64 threads]
Wait 30 seconds..............................
Results: 1693 seeks/second, 0.591 ms random access time (1103197231 < offsets < 65919945589567)

livecd ~ # ./seeker_baryluk  /dev/sdb 128
Seeker v3.0, 2009-06-17, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sdb [128749953024 blocks, 65919975948288 bytes, 61392 GB, 62866188 MB, 65919 GiB, 65919975 MiB]
[512 logical sector size, 512 physical sector size]
[128 threads]
Wait 30 seconds.............................
Results: 3099 seeks/second, 0.323 ms random access time (340085203 < offsets < 65919840239588)

I started stress testing the array which is likely what I will be doing up to the next couple of days. I just started not too long ago and already have quite a bit used:

Code: 
livecd ~ # df -H /jfs
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sdb1               66T   287G    66T   1% /jfs
livecd ~ # df -h /jfs
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1              60T  268G   60T   1% /jfs
livecd ~ #

One bad sign is with my sequential write tests earlier and with my stress testing I saw a timeout as well as several more:

Code: 
livecd ~ # ./cli64 event info | head -20
Date-Time            Device           Event Type            Elapsed Time Errors
===============================================================================
2010-10-10 12:00:47  Enc#2 SES2Device Time Out Error
2010-10-10 11:53:44  Enc#3 020        Time Out Error
2010-10-10 11:53:32  Enc#3 001        Time Out Error
2010-10-10 11:53:19  Enc#2 022        Time Out Error
2010-10-10 11:24:41  Enc#3 010        Time Out Error
2010-10-10 11:06:27  E2 Fan 02        Failed
2010-10-10 11:06:27  E2 Fan 01        Failed

Now from iostat output it looks like the pauses lasted only 5 seconds or so and thus I am not sure that its not just hitting bad sectors or not but things aren't looking so great. I have about 50 TiB of data scheduled to write and multi-threaded seeks set to run in a loop. I guess I will see if any of the devices goes to the fail state or actually becomes degraded or its only timeouts I am seeing...
 
Last edited:
It's a great card. Sequential R/W will hit a wall at 1GB/s (technically 960MBps) with a single SAS Wide uplink from expander to Areca card using SATA-II drives. That can double with two SAS expanders (or one SAS-2 expander) = 1920MBps potential with drives/arrays spread across both links. You shouldn't see any compatibility issues with those drives, btw. The headaches of the previous gen SAS cards with onboard expander are a thing of the past.

Good luck!
 
Last edited:
Looking good. I care more about rebuild times at this point since I only have quad gigabit NICs in my file server, so hopefully you can report in on that. Also, did you buy one of those last two T221s off eBay? :p
 
Well so far its looking good as no problems on the initialization of raid60. I wanted to first do some tests on raid0 just to push the card all the way but I found problems even accessing the block devices on linux due to even 2.6.35 having and old enough arcmsr driver that it would not detect the ARC-1880i.

All of the live-cd/usb's I had access to had the arcmsr driver built-in to the kernel so I couldn't load a newer version on it. Because this was not looking good and I wanted to atleast get some stress test going on the disks I deleted the raid0 array I had setup and racked another 24 disk machine and hooked it up to the other SFF-8087 port on the card and setup a raid60 array on it.

arc1880i_2.png


I later downloaded and run off a gentoo disk that had arcmsr as a module and was able to get the array up and going:

Code: 
livecd ~ # ./cli64 vsf info
  # Name             Raid Name       Level   Capacity Ch/Id/Lun  State
===============================================================================
  1 ARC-1880-VOL#000                 Raid60    80.0GB 00/00/00   Normal
  2 ARC-1880-VOL#003                 Raid60  65920.0GB 00/00/01   Initializing(7.9%)
  3 VOL#000R60Vol2-1 Raid Set # 000  Raid6     40.0GB 00/00/00   Normal
  4 VOL#000R60Vol2-2 Raid Set # 001  Raid6     40.0GB 00/00/00   Normal
  5 VOL#003R60Vol2-1 Raid Set # 000  Raid6   32960.0GB 00/00/01   Initializing(7.9%)
  6 VOL#003R60Vol2-2 Raid Set # 001  Raid6   32960.0GB 00/00/01   Initializing(8.1%)
===============================================================================
GuiErrMsg<0x00>: Success.

Code: 
livecd ~ # ./cli64 hw info
The Hardware Monitor Information
=====================================================
[Controller H/W Monitor]
  CPU Temperature  : 52 C
  Controller Temp. : 46 C
  12V              : 12.160 V
  5V               : 5.160 V
  3.3V             : 3.408 V
  DDR-II +1.8V     : 1.872 V
  CPU    +1.8V     : 1.872 V
  CPU    +1.2V     : 1.280 V
  CPU    +1.0V     : 1.072 V
  DDR-II +0.9V     : 0.944 V
  Battery Status   : 100%
[Enclosure#1 : ARECA   SAS RAID AdapterV1.0]
[Enclosure#2 : LSILOGICSASX36 A.1      7015]
  Fan 00           : 0 RPM
  Fan 01           : 0 RPM
  Fan 02           : 0 RPM
  Enclosure Temp   : 35 C
[Enclosure#3 : LSILOGICSASX36 A.1      7015]
  Fan 00           : 5340 RPM
  Fan 01           : 5250 RPM
  Fan 02           : 5280 RPM
  Enclosure Temp   : 33 C
=====================================================
GuiErrMsg<0x00>: Success.

Also cli64 seems to be much faster. I am not sure if its the controller or something else but cli64 vsf to check volume status reports much sooner than it did in the past as well as cli64 event info.

A couple of tests while initializing showed 500 megabytes/sec read/write which isn't great but it is doing a foreground initialization. I will do more testing once the array is up and running and after that try software raid0 over two raid0 sets to see what the limits of the card are.

The machine I am testing it on:

Code: 
livecd ~ # dmidecode | grep -i name | head  -1
        Product Name: X8DT3
livecd ~ # free -m
             total       used       free     shared    buffers     cached
Mem:         24159       1446      22713          0        158        817
-/+ buffers/cache:        470      23688
Swap:            0          0          0
livecd ~ # cat /proc/cpuinfo  | grep -i model\ name
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
livecd ~ #

Since each core is only 2.27 Ghz I am worried about DD being CPU bound for sequential read/write tests. Anyone know anything that uses less CPU usage?

I will be reporting rebuild times. so far the initialization is going decently fast considering I am doing two 24 disk raid6 arrays at once. Looks like its going to take around 24 hours. Initializing the 80 GB OS volume took 2 minutes 10 seconds.
 
houkouonchi said:
I later downloaded and run off a gentoo disk that had arcmsr as a module and was able to get the array up and going:
Click to expand...

So, did you download the latest arcmsr source code from areca's web site, compile it as a module, and use that driver?

What version of arcmsr were you using? Areca has a strange version numbering on arcmsr, it seems they keep the same version, 1.20.0X.15 on every revision, but just append a date code. Here are the last few entries from their changelog:

ftp://ftp.areca.com.tw/RaidCards/AP_Drivers/Linux/DRIVER/SourceCode/ReleaseNote.txt

05/10/2010 1.20.0X.15 1. support ARC-1880 series on kernel 2.6.X
07/27/2010 1. fix the system lock-up on Intel Xeon 55XX series, x86_64
07/29/2010 1.20.0X.15 1. revise the codes for scatter-gather applicable to RHEL5.0 to RHEL5.3
Click to expand...
 
john4200 said:
So, did you download the latest arcmsr source code from areca's web site, compile it as a module, and use that driver?
Click to expand...

That is exactly what I did.

Sadly even the newest kernel is using a driver from 2008 and no entries for the ARC-1880 it looks like:

Code: 
root@dekabutsu: 09:22 AM :/src# wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.35.7.tar.gz
--09:22:24--  http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.35.7.tar.gz
           => `linux-2.6.35.7.tar.gz'
Resolving www.kernel.org... 149.20.20.133, 204.152.191.37
Connecting to www.kernel.org|149.20.20.133|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 88,323,744 (84M) [application/x-gzip]

100%[==============================================================>] 88,323,744     5.24M/s    ETA 00:00

09:22:43 (4.51 MB/s) - `linux-2.6.35.7.tar.gz' saved [88323744/88323744]

root@dekabutsu: 09:22 AM :/src# tar xzvf linux-2.6.35.7.tar.gz | grep -i arcmsr
linux-2.6.35.7/Documentation/scsi/ChangeLog.arcmsr
linux-2.6.35.7/Documentation/scsi/arcmsr_spec.txt
linux-2.6.35.7/drivers/scsi/arcmsr/
linux-2.6.35.7/drivers/scsi/arcmsr/Makefile
linux-2.6.35.7/drivers/scsi/arcmsr/arcmsr.h
linux-2.6.35.7/drivers/scsi/arcmsr/arcmsr_attr.c
linux-2.6.35.7/drivers/scsi/arcmsr/arcmsr_hba.c
root@dekabutsu: 09:22 AM :/src# grep DRIVER_VERSION linux-2.6.35.7/drivers/scsi/arcmsr/arcmsr.h
#define ARCMSR_DRIVER_VERSION                "Driver Version 1.20.00.15 2008/11/03"
root@dekabutsu: 09:23 AM :/src# grep -ri 1880  linux-2.6.35.7/drivers/scsi/arcmsr/
root@dekabutsu: 09:23 AM :/src#

I will email areca and see if they can commit the newest driver to the linux kernel.
 
2.6.36rc7 has 2010-02-02 version of arcmsr, but that is still 6 months out of date. I'm not sure why areca seems to have difficulty submitting driver code to the kernel developers.

Code: 
$ wget -q http://www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.36-rc7.tar.bz2

$ tar jxvf linux-2.6.36-rc7.tar.bz2 | grep -i arcmsr
tar: Record size = 8 blocks
linux-2.6.36-rc7/Documentation/scsi/ChangeLog.arcmsr
linux-2.6.36-rc7/Documentation/scsi/arcmsr_spec.txt
linux-2.6.36-rc7/drivers/scsi/arcmsr/
linux-2.6.36-rc7/drivers/scsi/arcmsr/Makefile
linux-2.6.36-rc7/drivers/scsi/arcmsr/arcmsr.h
linux-2.6.36-rc7/drivers/scsi/arcmsr/arcmsr_attr.c
linux-2.6.36-rc7/drivers/scsi/arcmsr/arcmsr_hba.c

$ grep -i "driver version" linux-2.6.36-rc7/drivers/scsi/arcmsr/*
linux-2.6.36-rc7/drivers/scsi/arcmsr/arcmsr.h:#define ARCMSR_DRIVER_VERSION		     "Driver Version 1.20.00.15 2010/02/02"
 
war9200 said:
Hi,

How stable are the Areca cards in Linux?

Have you performed any load tests?

I and many others have these issues with 3ware cards (which I like the CLI/smart pass-through/etc) but the reset is annoying:

http://forums.storagereview.com/ind...oller-resets-under-load-on-linux/page__st__20
Click to expand...

I have had that problems on 3ware controllers including that issue. I think eventually the machine gets "Rejecting I/O to offline device" when it gets those timeouts.

I have also had major issues with high iowait (especially with parity raid) and extremely slow write speed as well on raid6 (20 megabytes/sec). Even with raid10 the performance of 3ware has been crap. I have seen several times where the array will never rebuild or just crap out when rebuilding (vs doing a rebuild loop). I have seen 3ware just randomly eat arrays as well.


We switched to LSI and had major issues with them as well. They would get the same timeouts and then 'Rejecting I/O to offline device' all the time. Almost always switching the controller would fix the issue and when it crashed its like the whole controller crashed as when you restarted it the machine would say cache data was lost and to check the BBU. The BBU shows the charge is fine which leads me to believe its faulty firmware or something causing the whole controller to crash and lose its data.

We finally switched to areca and currently have over 450 machines using areca controllers. We have seen timeout issues on these as well but so far they have all been attributed to failing drives... IE: I see the same port timing out and physicaly viewing the machine I can see the light sticking on, stuff like that. Looking at the event history will pretty much always show a timeout on one of the drives and if left in I will see it over and over until the drive eventually fails or someone pulls it/replaces it.

Several times I look at the disk info to see that the reallocation count has gone below the SMART threshold so I am not really sure why the controller is not kicking these disks out of the array (when it gets to that point) and this is another defect of areca controllers.

I know the BBU is working properly though as areca tells you when it has to use the BBU to restore cache data as it will tell you that it powered on with the BBU and we haven't seen the same type of FS corruption on areca that we have had on these LSI machines that ran into timeouts often.

Here is an example:

On console output I see:

Code: 
arcmsr0: abort device command of scsi id = 0 lun = 1
arcmsr0: abort device command of scsi id = 0 lun = 1
arcmsr0: abort device command of scsi id = 0 lun = 1
arcmsr0: abort device command of scsi id = 0 lun = 1

Its power cycled and in event history I see something like this:

Code: 
Date-Time Device Event Type Elapsed Time Errors

============================================================================
===
2010-10-01 14:38:07 H/W MONITOR Power On With Battery Backup
2010-10-01 13:53:08 Enc#1 Slot#2 Time Out Error
2010-10-01 10:02:59 SW API Interface API Log In
2010-09-30 22:27:46 Enc#1 Slot#2 Time Out Error
2010-09-30 22:27:36 Enc#1 Slot#2 Time Out Error
2010-09-30 22:27:26 Enc#1 Slot#2 Time Out Error
2010-09-30 22:27:15 Enc#1 Slot#2 Time Out Error
2010-09-30 22:26:40 Enc#1 Slot#2 Time Out Error
2010-09-30 22:26:30 H/W MONITOR Power On With Battery Backup
2010-09-30 22:25:36 Enc#1 Slot#2 Time Out Error
2010-09-30 22:25:26 Enc#1 Slot#2 Time Out Error
2010-09-30 22:25:11 Enc#1 Slot#2 Time Out Error
2010-09-30 22:25:01 Enc#1 Slot#2 Time Out Error
2010-09-30 22:24:20 Enc#1 Slot#2 Time Out Error
2010-09-30 22:24:10 Enc#1 Slot#2 Time Out Error
2010-09-30 22:23:51 Enc#1 Slot#2 Time Out Error
2010-09-30 22:23:41 Enc#1 Slot#2 Time Out Error
2010-09-30 22:23:31 H/W MONITOR Power On With Battery Backup
2010-09-30 21:17:50 Enc#1 Slot#2 Time Out Error
2010-09-30 15:55:14 Enc#1 Slot#2 Time Out Error
2010-09-30 12:26:08 Enc#1 Slot#2 Time Out Error
2010-09-30 12:25:58 Enc#1 Slot#2 Time Out Error
2010-09-30 10:07:56 Enc#1 Slot#2 Time Out Error
2010-09-29 19:43:17 Enc#1 Slot#2 Time Out Error
2010-09-29 18:22:48 Enc#1 Slot#2 Time Out Error
2010-09-29 18:22:38 Enc#1 Slot#2 Time Out Error
2010-09-29 18:22:28 Enc#1 Slot#2 Time Out Error
2010-08-13 01:47:32 H/W MONITOR Raid Powered On
2010-08-12 16:31:11 H/W MONITOR Raid Powered On

When doing disk info all the drives are fine but the single drive shows the sector re-allocation count way below the threshold:

Code: 
# cli64 disk info drv=1
Drive Information
===============================================================
Device Type : SATA(5001B4D40F7D1010)
Device Location : Enclosure#1 Slot#1
Model Name : ST31000528AS
Serial Number : 9VP4XXX
Firmware Rev. : CC38
Disk Capacity : 1000.2GB
Device State : NORMAL
Timeout Count : 0
Media Error Count : 0
SMART Read Error Rate : 111(6)
SMART Spinup Time : 96(0)
SMART Reallocation Count : 100(36)
SMART Seek Error Rate : 72(30)
SMART Spinup Retries : 100(97)
SMART Calibration Retries : N.A.(N.A.)
===============================================================
GuiErrMsg<0x00>: Success.
# cli64 disk info drv=2
Drive Information
===============================================================
Device Type : SATA(5001B4D40F7D1011)
Device Location : Enclosure#1 Slot#2
Model Name : ST31000528AS
Serial Number : 9VP4XXX
Firmware Rev. : CC38
Disk Capacity : 1000.2GB
Device State : NORMAL
Timeout Count : 0
Media Error Count : 0
SMART Read Error Rate : 97(6)
SMART Spinup Time : 96(0)
SMART Reallocation Count : 2(36)
SMART Seek Error Rate : 72(30)
SMART Spinup Retries : 100(97)
SMART Calibration Retries : N.A.(N.A.)
===============================================================
GuiErrMsg<0x00>: Success.

I think a lot of the problem is that we are using non-enterprise disks. I have yet to see this on machines where we use NS or SS (SAS) seagate disks. I believe the timeout is caused by drives running into bad sectors and doing re-allocations. I think the controller handles it ok most of the time but when it has to do it too many times in a row the OS times out the controller while its waiting on writing to the drive. Of course when one of the SMART values get lower than the threshold it is my oppinion the controller should fail the drive.

I am trying to get areca to make a method to kick the drive out of the array remotely (both LSI/3ware support this). Their enterprise customers really need this feature as almost half of our machines are in a data center that doesn't have our own on-site staff on and we generally don't want the remote hands messing with the drives so if the problem is causing a major stability issue we have to make a special trip to the other DC that is 16 miles from our main one just to pull a drive.

In the end Areca has shown itself to be FAR less problematic than both LSI and 3ware not to mention perform the best out of the three. We do shared hosting with machines under high I/O activity 24/7 and 3ware/LSI really didn't seem to handle being maxed out all the time very well. I think everyone here is happy we switched to Areca.

I have had to recover arrays three times I think from areca. Two were pretty easily recovered. The third was raid10 and the wrong two disks failed so nothing could be done about that (not the controllers fault). We have also had file-system corruption on one (well an fsck that would never finish). Not totally sure if that is controller related but I think enabling disk cache was a mistake as I originally thought that meant write back when it means the buffer on the disk which can cause corruption on a power loss even with a BBU. This is within a pool of 450 areca machines. In the 300 LSI machines we have probably replaced 40 controllers (I am not exaggerating) and lost several arrays a couple times on LSI but the biggest problem has been file-system corruption and server stability. We have probably lost 15-20 machines worth of file-systems on LSI controllers. In about 80 3ware machines we probably had to replace 5-10 of the controllers and 10-15 arrays or the arrays got in a state where they would never rebuild and data had to be offloaded.
 
You must log in or register to reply here.
Back
Top