understand mpt2sas kernel messages?

[SirMaster]

Does anyone have experience understanding these messages that I get in my kernel log from time-to-time?

mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)

It happens only under heavy write loads like when I dd /dev/zero to my zpool so all disks write at once as fast as they can.

I blame the SAS expander that I got recently, but I'm just wondering if I can find out what the messages mean and If I can just ignore them. I have not seen any negative effect from the messages but I just like to find out what I can.

Thanks.

 

[Snowknight26]

linux/drivers/message/fusion/lsi/mpi_log_sas.h:

/****************************************************************************/
/*  IOC LOGINFO defines, 0x00000000 - 0x0FFFFFFF                            */
/*  Format:                                                                 */
/*      Bits 31-28: MPI_IOCLOGINFO_TYPE_SAS (3)                             */
/*      Bits 27-24: IOC_LOGINFO_ORIGINATOR: 0=IOP, 1=PL, 2=IR               */
/*      Bits 23-16: LOGINFO_CODE                                            */
/*      Bits 15-0:  LOGINFO_CODE Specific                                   */
/****************************************************************************/

#define IOC_LOGINFO_ORIGINATOR_PL                       (0x01000000)
#define PL_LOGINFO_CODE_RESET                                (0x00110000)
#define PL_LOGINFO_SUB_CODE_SATA_LINK_DOWN                   (0x00000D00)
#define PL_LOGINFO_SUB_CODE_OPEN_FAIL_NO_DEST_TIME_OUT       (0x00000001)

Therefore:

0x31110d01 == (
MPI_IOCLOGINFO_TYPE_SAS |
IOC_LOGINFO_ORIGINATOR_PL |
PL_LOGINFO_CODE_RESET |
PL_LOGINFO_SUB_CODE_SATA_LINK_DOWN |
PL_LOGINFO_SUB_CODE_OPEN_FAIL_NO_DEST_TIME_OUT
)

Could be a bad cable.

http://serverfault.com/questions/407703/deciphering-continuing-mpt2sas-syslog-messages

 

[SirMaster]

Thanks for mapping the error number out for me.

I've swapped out absolutely everything in my chain and they still happen sometimes so I think I'm just going to have to ignore them.

My normal chain goes:

Onboard LSI 2308 -> reverse breakout cable -> Chenbro CK12803 Expander -> sas cables -> Norco 4020 backplanes.

I started out by swapping out the onboard LSI 2308 with an LSI 2008 discrete card and connecting it to the expander with a normal sas cable thus eliminating the onboard LSI and reverse breakout from the equation, but still the messages.

Then I swapped out the expander with another identical Chenbro CK12803 yet still got the errors.

Then I took all the disks out of the backplanes and connected them with forward breakout cables directly to the expander, yet still errors.

The only constant in my tests are the disks which I can't really test?

Perhaps I will rotate the disks into onboard SATA to see if I can pinpoint which disk(s) are perhaps causing this?

Otherwise what might it be? Just random software conflicts in using "cheap" SATA disks on SATA hardware and an older cheap sas expander model in general?


What do you think about adjusting disk timeouts to resolve this?

How about the risk in ignoring these messages? If it just affects performance then so be it. My performance is excellent IMO so that's not an issue.


I did try some tests with no expander by connecting 8 disks to the onboard 2308 and the other 4 disks to the 2008 card and I did not get errors yet. This leads me to believe it's the expander, but again, I have two of the expanders which both cause the messages, so it's like its that expander model itself, not that its defective exactly.

 

[SirMaster]

Nevermind all that. I finally found out the issue.

1 of my disks was bad. I mean it appears to work fine, passes SMART tests and everything, but it was causing these SATA link resets.

I noticed that on my LSI this 1 drive was always only negotiating at 3G instead of 6G and when I removed that single drive from my SAS chain I no longer get those messages after hours and hours of stress testing.

 

[Stanza33]

was the drive actually bad....

or was it crippled on NCQ ?

eg

Os thinks the drive has an NCQ value of 32 (most drives these days).... but drive actually has an NCQ of 1.

hence

as I have found out with a bunch of similar drives (seagate STblahblahM001)

2 out of the 12 that I have are oddballs.... in that 10 of the drives announce as having NCQ depths of 32.... but the "special" drives only come up as NCQ depths of 1

so maybe the drive is ok, just the OS / controller / expander or driver thinks they have a greater NCQ depth.... and throws up errors at full speed?

.

 

[mwarps]

Had a ton of 0x11s due to shitty Dell OEM drives a few years back. Feel your pain.

Failure mode began with the resets and progressed rapidly to being marked bad from there and then ZFS starting to get angry due to dropped drives.

Replacement - 'now would be a good time'

 

[SirMaster]

was the drive actually bad....

or was it crippled on NCQ ?

eg

Os thinks the drive has an NCQ value of 32 (most drives these days).... but drive actually has an NCQ of 1.

hence

as I have found out with a bunch of similar drives (seagate STblahblahM001)

2 out of the 12 that I have are oddballs.... in that 10 of the drives announce as having NCQ depths of 32.... but the "special" drives only come up as NCQ depths of 1

so maybe the drive is ok, just the OS / controller / expander or driver thinks they have a greater NCQ depth.... and throws up errors at full speed?

.

Well considering I have 6 drives that are all the same model/firmware shouldn't they all support the same NCQ? All 6 are at 32.

1 of the 6 drives would only ever negotiate at 3Gb on the SAS interface while the other 5 were always at 6Gb. Shouldn't that also be the same across identical models/firmware?

If you have 6 things and 5 are identical and one is different (but is the same model/firmware) I would call that defective. I plan to RMA the disk, It's being resilvered out of my array as I write this.