SAS Expander Usage

[ixeous]

This is really a knowledge share about a problem that I experienced and found little useful information on resolving the issue. I've posted in another forum also, but this one seems to come up regularly in google so it seems like a good place to post the info.

I built a server with 24 data drives and 2 internal OS drives. The server used onboard SATA for the OS drives. The 24 data drives were driven via a LSI 9211-8i (LSI2008 chip) controller and an Intel RES2V240 (LSISAS2X24 chip) SAS expander. The 24 data drives are all 4TB SATA drives (HGST H3IK40003272SP 0S03355). Intel's documentation of the expander card says that it is compatible with Intel Raid Controller RS2WC080 which uses the LSI2008 chip. No compatibility issues should exist between the expander and the controller used. Both the controller and the expander were updated to the most recent firmware/bios. The server is running CentOS 6.3, fully patched, most recent CentOS kernel.

The controller has 2 8087 ports on the card. The expander had 6 8087 ports. One port on the controller and one port on the expander are used when connecting to controller to the expander. This leaves a maximum of 20 drives (5 8087 ports) on the expander.

The original configuration of drives/controller/expander was

HDDx4<===>9211<===>RES2V240<===>HDDx20

From the 24 data drives, 3 Linux md RAID6 devices (md1, md2, md3) were created (no hardware RAID) which were combined into a single LVM volume group and logical volume (mounted to /bigdata). md1 used the 4 drives connected directly to the controller and 4 drives connected to the expander. The other 2 md devices used all drives connected to the expander.


THE PROBLEMS:

SYMPTOM 1:
Log files would have repeated entries of

kernel: sd 0:0:13:0: attempting task abort! scmd(ffff88004a19d8c0)
kernel: sd 0:0:13:0: task abort: SUCCESS scmd(ffff88004a19d8c0)
kernel: sd 0:0:13:0: attempting task abort! scmd(ffff88004a19ddc0)
kernel: sd 0:0:13:0: task abort: SUCCESS scmd(ffff88004a19ddc0)
kernel: sd 0:0:13:0: attempting task abort! scmd(ffff88001c7d29c0)
kernel: sd 0:0:13:0: task abort: SUCCESS scmd(ffff88001c7d29c0)

The address 0:0:X:0 indicates the specific drive having the problem. The X was always a number associated with one of the 20 drives on the expander, but never on the controller.

There were no log entries related to mpt2sas (the controller driver).

These errors only show up when I/O is being performed on the drives such as RAID resync (SYMPTOM 2) or reading/writing data in production. This system is a backup to disk system so production use means heavy-ish I/O during backups. The errors occurred without fail.


SYMPTOM 2:
The RAID devices md2 and md3 would automatically resync periodically. All drives were present and functional via mdadm and mdstat. The RAID device md1 never had a problem.


SYMPTOM 3:
The smartd/smartctl tools would initially detect all drives and that all drives were SMART capable, but would later claim the device is not capable

Device: /dev/sdm [SAT], is SMART capable. Adding to "monitor" list.

Then

Device: /dev/sdm [SAT], not capable of SMART self-check


DUPLICATING THE PROBLEM:
I was able to repeat the problem using dd to place I/O load on the system. All dd commands ran simultaneously.

# dd if=/dev/zero of=/bigdata/ddoutfile bs=4096 count=100000000 &
# dd if=/dev/md3 of=/dev/null &
# dd if=/dev/md2 of=/dev/null &
# dd if=/dev/md1 of=/dev/null &

After a few minutes, SYMPTOM 1 appeared in the log files and continued to appear every minute or two.


RECONFIGURE:
The issue only occurred during I/O load, and only affected drives on the expander. I added a second expander and split the drives evenly between them.

HDDx12<===>RES2V240<===>9211<===>RES2V240<===>HDDx12

Since doing this, I have not been able to duplicate the problem with dd and the other symptoms have not occurred as of yet.

CONCLUSIONS:
It seems that 20 SATA 3 drives configured as RAID6 devices exceeds the I/O capabilities of the controller/expander combination. Using 12 SATA 3 drives does not seem to cause problems. There are still many unknowns:

1. Is the problem the expander or the 8087-8087 controller to expander connection?
2. Would using SATA 2 drives have the same issue?
3. Does the read/write process of RAID6 parity calculation contribute to the problem?
4. What is the “magic” number of drives that cause the problem? 14? 16? 18?

I will not be using expanders in future builds. I intended to use the expander to lower the cost of high disk count servers. Using 1 controller and 1 expander was cheaper than using 3 controllers, but 3 controllers is cheaper than 1 controller and 2 expanders. I have other production systems that use 3 controllers and have not experienced any problems.

 

[patrickdk]

SATA2 should double the issue, and limit you to 8 disks.

The magic number will change depending on the disks, 15/16 likely in this case.

The issue is the STP, since the hba -> expander can't talk sata, the expander has to take the sata infomation, wrap it up, and pass it along. There are countless known issues with this in zfs, where this usecase seems to be very common.

I would expect not using the expander, or using disks that talk sas would solve all issues.
I don't know of anyone that has solved this, as there seems to be too many moving parts.
Could be expander firmware, could be hba driver, maybe hba but doesn't sound like it in this case.

 

[odditory]

It seems that 20 SATA 3 drives configured as RAID6 devices exceeds the I/O capabilities of the controller/expander combination. Using 12 SATA 3 drives does not seem to cause problems. There are still many unknowns:

Correction - your particular controller/expander/motherboard/backplane/host driver/operating system/cable/cable length/powersupply combination. If you'd been able to reproduce the issue with for example an HP SAS Expander in single and dual linked configurations then we could discuss what else it might be - there are at least half a dozen other variables I can think of that you didn't appear to factor based on the writeup anyway. I realize there's only so much testing you can do before a point comes you just don't care anymore but the testing methodology was very narrow.

And if this is for business/enterprise use that's a whole separate discussion -- cheap SATA disks on lowcost SAS controllers and expanders have been popularized within the enthusiast demographic for specific purposes, like home media storage. Its not a guarantee or endorsement that its the best practice for a business setting.

 

[omniscence]

I use the same combination of controller and expander, although in a dual-linked configuration with 16 SATA disks. The disks are a mixture of 4x 3 TB WD REDs, 8x 2 TB Samsung F4EG and 4x 2 TB Samsung F3EG. During the last mdadm raid check, where all disks are read simultaneously (there are 3 different arrays on these disks), I saw the same messages like your symptom 1, altough for all disks. Neither the RAID layer complained nor the filesystem, so I guess no corruption occured (also it were mostly reads during that time). If have not been able to replicate that issue, nor did I try very hard.

I always was wondering how SAS 6 Gbps expanders handle SATA 3 Gbps traffic. There are basically 3 methods that come to my mind:
- cache the slow packets from the drive and send them upstream at the higher speed once complete (I doubt that, it would require a lot of cache for larger systems)
- send the packets upstream at the same speed they arrive, effectively slowing the upstream connection to 3 Gbps (would that slow down a single lane, a "wide" lane or the whole double-link?)
- send the packets upstream at the same speed, but interleave the packets with packets from other slow drives (most effective, but if one drive misbehaves, the whole link is affected)

I would appreciate if someone more knowledgeable could shed some light on this issue.

EDIT: Well, I just did some quick tests. While I could not trigger symptom 1, I now have a better understanding of the performance questions. I removed one of the links between controller and expander. While the performance of all three arrays read individually sum up to 1200 MB/s, if read simultaneously they yield less than 600 MB/s. If the SATA3 and an SATA2 array are read in parallel I could get at least 800 MB/s. Hence, the throughput of the 6 Gbps SAS 4x link is less than a half "wide" 3 Gbps 4x link if mostly SATA2 drives are used. Since I only have 4 SATA3 drives at hand I cannot test at which bandwidth the expander saturates for a 6 Gbps-only configuration. I could however throw some SSDs in the mix...

 

[patrickdk]

The expander works in mode 2 normally, the speed of the whole thing, goes the speed of the slowest link speed, so everything would operate at 3g.

Most all lsi stuff supports multiplexing, so they could support your 3rd idea, though it's normally disabled. I don't know of any expander that currently supports it, though the intel expander lsi chipset does, it was disabled.

 

[ixeous]

SATA2 should double the issue, and limit you to 8 disks.

The magic number will change depending on the disks, 15/16 likely in this case.

The issue is the STP, since the hba -> expander can't talk sata, the expander has to take the sata infomation, wrap it up, and pass it along. There are countless known issues with this in zfs, where this usecase seems to be very common.

This is the info that I had not found previously. I saw a number of posts related to zfs and the configuration, but nothing that indicated that using fewer drives on the expander would "solve" the issue.