External SAS oddness

[lizardking009]

I have an Adaptec 5085 (8-port external) connected to an EnhanceRack R8-MS with two SFF-8088 cables. The R8-MS is as dumb as a JBOD can be, consisting of an 8 slot hotswap case with two SFF-8088 -> 4 SATA breakout adapters. No expanders or anything else remotely interesting or intelligent.

I also have an Adaptec 5805 card attached to a SuperMicro barebones with 6 hotswap slots. When my boss assembled this box, he used cables without SGPIO so the SM backplane is as dumb as the EnhanceRack. There's no enclosure management.

Here's where it gets odd:

5085: in the Adaptec Storage Manager, it shows 8 drives connected. They're all listed under the heading "Direct attached". Each drive is assigned a PHY number. The PHY number does not correspond to the drive slot. However, if I get the device properties, it indicates the SCSI ID number which does correspond to the correct slot. ARCCONF provides similar correct information. The BIOS provides the PHY numbers.

The Adaptec manual indicates that PHY entries are related to expanders, but as far as I know, I don't have one in the mix at all. There certainly isn't one in the EnhanceRack.

This strange seemly cosmetic defect makes hit hard to tell at a glance which drive on the EnhanceRack is problematic since the PHY doesn't match to the slot.

5805: In the Adaptec Storage Manager, it shows 6 drives connected, 3 to each SFF-8087 connector. Those SFF-8087 connectors show as CN0 and CN1 and under each CN heading, there are 3 devices: 0, 1, and 2. Each device corresponds to the appropriate slot in the case. Perfectly logical.

Can anyone explain to me why the 5085 and 5805 behave so differently with regards to direct attached drives? Can anyone suggest how to make the 5085 behave like the 5805?

Does anyone know if the LSI external SAS or Dell Perc 6/e have the same kind of strangeness with dumb external JBODs?

Thanks.

 

[AreEss]

It's Adaptec. If you wanted predictability and consistent behavior, you should have smashed them with a hammer before installing them.

LSI 8480 does not exhibit any "strange" behaviors. It's also external only, so don't forget that. It works very well, and it works very fast. In fact, if you want performance, your ONLY choice is the LSI 8480. There is nothing comparable on the market, nor will there be any time soon.

The Supermicro also has SES (SAS Environmental Services) which goes across the data lines, so removing SGPIO is utterly pointless and a waste. It's also required because it's what handles removal and arrival notifications, since you have to use the backplane's local PHY. (The Adaptec side PHYs are just a connection to another PHY, basically.) This also ensures slots map correctly.
The Enhance is either cabled incorrectly, or it's trash. I have not tested the R8-MS. When going "just JBOD" there is still a requirement for a PHY, because of arrival/removal notifications and power control. Reverify that the breakouts are cabled correctly. If they are and it still doesn't match up, there's broken logic hiding on the backplane itself somewhere and you'll need a different chassis.

 

[lizardking009]

It's Adaptec. If you wanted predictability and consistent behavior, you should have smashed them with a hammer before installing them.


LSI 8480 does not exhibit any "strange" behaviors. It's also external only, so don't forget that. It works very well, and it works very fast. In fact, if you want performance, your ONLY choice is the LSI 8480. There is nothing comparable on the market, nor will there be any time soon.

Believe me, we're done with Adaptec. I've had many a bad day getting those cards to work with ESX. Its better on Windows/Hyper-V, but it was still a nightmare getting them to play nice with the SM white boxes.

My new SM barebones includes an LSI 1078 based UIO card. I am hoping its a lot less quirky and just "works", kind of like a PERC in a Dell. Still waiting on RAM and drives, so I won't know for a bit.

The Supermicro also has SES (SAS Environmental Services) which goes across the data lines, so removing SGPIO is utterly pointless and a waste. It's also required because it's what handles removal and arrival notifications, since you have to use the backplane's local PHY. (The Adaptec side PHYs are just a connection to another PHY, basically.) This also ensures slots map correctly.

On the SM without SGPIO, I have successfully replaced all the drives in a live array and built a LUN on the free space. The ASM didn't complain and showed which drives were removed. It was all consistent and worked fine. Not the way I'd do it, but it works so I'm not going to bother with it.

The Enhance is either cabled incorrectly, or it's trash. I have not tested the R8-MS. When going "just JBOD" there is still a requirement for a PHY, because of arrival/removal notifications and power control. Reverify that the breakouts are cabled correctly. If they are and it still doesn't match up, there's broken logic hiding on the backplane itself somewhere and you'll need a different chassis.

The SCSI IDs matched to the drive slots. The SCSI IDs were visible through ARCCONF and the Drive Properties in ASM. The PHYs didn't match and were visible through the BIOS and on the ASM main screen. Also, I tried an Areca 1221x (SATA only) with the JBOD and all the slots matched to the port numbers. I think the unit is cabled properly.

Other than that, the unit does work properly. Guess its just Adaptec being quirky. I wish I had an LSI external card to see if it behaves properly.

 

[AreEss]

Believe me, we're done with Adaptec. I've had many a bad day getting those cards to work with ESX. Its better on Windows/Hyper-V, but it was still a nightmare getting them to play nice with the SM white boxes.

Adaptec's been trash for many years, so yeah, no surprise there. I haven't touched one that isn't labeled "ServeRAID" in some years, and I'm much happier for it.

My new SM barebones includes an LSI 1078 based UIO card. I am hoping its a lot less quirky and just "works", kind of like a PERC in a Dell. Still waiting on RAM and drives, so I won't know for a bit.

Hopefully not like a PERC in a Dell, because PERC sucks. (Sorry, that's just how it is. They do.) I can't comment on the UIO extensively though, as I haven't used it. However, the 1078 based stuff should basically just be a standard LSI in a UIO form factor. It's just a PCI-Express slot with a proprietary physical.

On the SM without SGPIO, I have successfully replaced all the drives in a live array and built a LUN on the free space. The ASM didn't complain and showed which drives were removed. It was all consistent and worked fine. Not the way I'd do it, but it works so I'm not going to bother with it.

Yeah, SES is a godsend in so many ways. I'd cable SGPIO in the future, it's really worth it.

The SCSI IDs matched to the drive slots. The SCSI IDs were visible through ARCCONF and the Drive Properties in ASM. The PHYs didn't match and were visible through the BIOS and on the ASM main screen. Also, I tried an Areca 1221x (SATA only) with the JBOD and all the slots matched to the port numbers. I think the unit is cabled properly.

Other than that, the unit does work properly. Guess its just Adaptec being quirky. I wish I had an LSI external card to see if it behaves properly.

... okay, then yeah, I take back what I said about the Enhance. It's definitely not the problem. Glad you had the Areca around!
The problem is one I haven't seen in a while on Adaptec, but that's for lack of trying I suspect. It used to be not at all uncommon for Adaptrash to ignore SCSI IDs and quite literally map things completely randomly. It sounds to me like that's exactly what's going on here; Adaptec's just completely ignoring SCSI ID for mapping purposes. Sigh. Yeah. There's no fix for that, either. There never has been, since Adaptec insists on sticking to a decade old code base.

 

[lizardking009]

Hopefully not like a PERC in a Dell, because PERC sucks. (Sorry, that's just how it is. They do.) I can't comment on the UIO extensively though, as I haven't used it. However, the 1078 based stuff should basically just be a standard LSI in a UIO form factor. It's just a PCI-Express slot with a proprietary physical.

The PERC6 series is also 1078 based. I have a few of them around and can't complain. They're unremarkable, which is just how I like production hardware to be.

Yeah, SES is a godsend in so many ways. I'd cable SGPIO in the future, it's really worth it.

I totally agree. The whiteboxes I build have SPGIO connected.

... okay, then yeah, I take back what I said about the Enhance. It's definitely not the problem. Glad you had the Areca around!

I had actually bought the Areca to test a problem with dropping drives. I had been cursing the Enhance as it was a spur of the moment purchase (not my idea). It turned out to be the crappy 7200.11 ES.2 series Seagates, even with the SN06 firmware, they'd all drop from both the Areca and the Adaptec. I have some WD RE3 series drives in there now with the Adaptec and have been hammering it for two weeks without problems.

The problem is one I haven't seen in a while on Adaptec, but that's for lack of trying I suspect. It used to be not at all uncommon for Adaptrash to ignore SCSI IDs and quite literally map things completely randomly. It sounds to me like that's exactly what's going on here; Adaptec's just completely ignoring SCSI ID for mapping purposes. Sigh. Yeah. There's no fix for that, either. There never has been, since Adaptec insists on sticking to a decade old code base.

So this really is an example of Adaptec quirkiness....well I feel better about trying EnhanceRack + LSI. The Enhance is pretty nice for a dumb box. It feels well constructed and has stiff drive trays.

 

[AreEss]

The PERC6 series is also 1078 based. I have a few of them around and can't complain. They're unremarkable, which is just how I like production hardware to be.

Sure, but Dell's custom bastardized firmware. Major turnoff for me, when I can get the real deal direct from LSI.

I had actually bought the Areca to test a problem with dropping drives. I had been cursing the Enhance as it was a spur of the moment purchase (not my idea). It turned out to be the crappy 7200.11 ES.2 series Seagates, even with the SN06 firmware, they'd all drop from both the Areca and the Adaptec. I have some WD RE3 series drives in there now with the Adaptec and have been hammering it for two weeks without problems.

Yeah, Seagate's on my legendary Vendor Permanent Blacklist. This is their third strike, and it's so widespread, there's absolutely no way I can sell them without having to flat out lie to customers. I cannot even guarantee that they will function after one power cycle. I can't honestly sell something that unpredictable and unreliable; I'd have screaming customers and be eating dozens of drives.

So this really is an example of Adaptec quirkiness....well I feel better about trying EnhanceRack + LSI. The Enhance is pretty nice for a dumb box. It feels well constructed and has stiff drive trays.

Yeah, that's why I was a little hesitant to blame the Enhance. I've never had problems with them in the past, and I've used them for years. I've used a lot of their other products though, but none of the external racks. I'm not surprised at Adaptec being craptastic in the end; it's what they excel at. That also means I can add Enhance's externals to my limited support enclosure list! (I normally require environmental services at a minimum, so.)

 

[lizardking009]

Sure, but Dell's custom bastardized firmware. Major turnoff for me, when I can get the real deal direct from LSI.

Yeah, I wouldn't buy them outside of a Dell PowerEdge, but when they're inside one of those, they do fine. Completely unremarkable and fully supported.

Yeah, Seagate's on my legendary Vendor Permanent Blacklist. This is their third strike, and it's so widespread, there's absolutely no way I can sell them without having to flat out lie to customers. I cannot even guarantee that they will function after one power cycle. I can't honestly sell something that unpredictable and unreliable; I'd have screaming customers and be eating dozens of drives.

I deserve the Seagate blame at my work. I had and still have 12 7200.10 series ES drives running beautifully on my home server. No problems of any kind. I assumed the 7200.11 drives would be decent. Wow. Should have stuck with WD.

Yeah, that's why I was a little hesitant to blame the Enhance. I've never had problems with them in the past, and I've used them for years. I've used a lot of their other products though, but none of the external racks. I'm not surprised at Adaptec being craptastic in the end; it's what they excel at. That also means I can add Enhance's externals to my limited support enclosure list! (I normally require environmental services at a minimum, so.)

How do you like Enhance's iSCSI stuff? I've been eyeballing the T8-IP... its a cheap tower iSCSI unit with redundant PSU... a rare beast to be sure.

I prefer environmental services as well, but the boss wanted cheap and wanted now. The Enhance was both. If I had to order a rack JBOD now, I'd be eyeing either the SM 16 bay with expander and JBOD kit or the Enhance JS16.

 

[simplesam]

Sure, but Dell's custom bastardized firmware. Major turnoff for me, when I can get the real deal direct from LSI.

If you flash the Perc 6 with the latest LSI firmware, then use the latest LSI drivers, the performance of the perc 6 improves.

I have two perc6e/512mb, identified by windows as 8888elp's.

 

[lizardking009]

If you flash the Perc 6 with the latest LSI firmware, then use the latest LSI drivers, the performance of the perc 6 improves.

I have two perc6e/512mb, identified by windows as 8888elp's.

I don't doubt that it works, but I hope you didn't try it on a supported system! Dell support wouldn't support it.

 

[AreEss]

Yeah, I wouldn't buy them outside of a Dell PowerEdge, but when they're inside one of those, they do fine. Completely unremarkable and fully supported.

Yeah, but I wouldn't sell or recommend a PowerEdge to anyone anyways. Dell has hidden minimum yearly purchases when you cross a certain point; otherwise, they flat out refuse to keep any parts or people local, and you get "maybe we'll be there in 4 hours - maybe we'll be there in 4 days" support. (I've dealt with three shops that have had this problem, all of them threw out Dell afterward.)

I deserve the Seagate blame at my work. I had and still have 12 7200.10 series ES drives running beautifully on my home server. No problems of any kind. I assumed the 7200.11 drives would be decent. Wow. Should have stuck with WD.

I don't sell WDC either, because of their RMA policy. (Any drive returned from RMA used to get a 30 day warranty, period. New, refurb, didn't matter - 30 days.) They're also a pain if you need to do bulk RMAs. I only deal with Samsung and Hitachi in the SATA space at this point.
Didn't hurt that I read the IBM warning on the Seagate rebrands; certain firmwares got a note that any power cycling or hotplug could cause the drives to become completely inaccessible and that a firmware fix would not be available for months. Explained the very large numbers of DOAs I got, too.

How do you like Enhance's iSCSI stuff? I've been eyeballing the T8-IP... its a cheap tower iSCSI unit with redundant PSU... a rare beast to be sure.

To be blunt, iSCSI just isn't reasonable without overpriced HBAs and an entirely separate network. I don't sell it actively, and generally do my best to steer people away from it. It's just too immature, unstable, and unpredictable to be used in production.

I prefer environmental services as well, but the boss wanted cheap and wanted now. The Enhance was both. If I had to order a rack JBOD now, I'd be eyeing either the SM 16 bay with expander and JBOD kit or the Enhance JS16.

My personal favorite for a 16 comes from AIC Industrial. (DISCLAIMER: I've been using AIC for years, and they were and are my exclusive provider for chassis for 4 and 8 way Opterons.) The entire XJ line is quite nice, and comes in enough flavors to please anyone. 2U with 24 bays, 2U with 12, 3U with 16, dual I/O modules, dual PSU, the works. They're excellent chassis and frequently OEM'd with good reason.

 

[simplesam]

I don't doubt that it works, but I hope you didn't try it on a supported system! Dell support wouldn't support it.

I am able to flash it back to a dell firmware, if needed. It's just a case of trying to get more performance, at a lower price.