HP SAS Expander Owner's Thread

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
Well I bet you could reverse it too (x4->x8 for expander & x8 for the radi card, which is probably better since you'd want the PCIe bandwidth for the raid card not the expander.

Since I had no desire to deal with that, I bought a P6T7 so everything is a physical x16 slot :) It may seem dumb but a few of those converters covers the price differential between that an a lower-end X58 board.
 

nitrobass24

[H]ard|DCer of the Month - December 2009
Joined
Apr 7, 2006
Messages
10,462
odditory, are you running Hitachi's?

Have you happened to try any Seagate LPs?

Any other updates from your testing?
 

nitrobass24

[H]ard|DCer of the Month - December 2009
Joined
Apr 7, 2006
Messages
10,462
Also the external port, is that for input or output to an external box?
Does anyone have a link for a manual for this thing?
 

BENN0

Weaksauce
Joined
Nov 23, 2009
Messages
95
Search hp.com for "sas expander card". The problem is all documentation and specifications are based on capabilities and limitations of the HP servers this card is officially supported in.
For example the documentation mentions a 12TB logical drive limit (and somewhere else it is 14TB) but this is probably based on the fact that the largest HP server this card supports only can hold 12 (or 14) drives and HP's largest drive is currently a 1TB SATA drive.

The only way we can figure out the real capabilities and limits of this card is by sharing our findings here.
 

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
It works great with my 5085. No problems at all.

Mine lights up with green lights, but the 5805 doesn't see it at all :-( Time to troubleshoot. And yes it's a green PCB version with an 11/2009 manufacture date.
 

BENN0

Weaksauce
Joined
Nov 23, 2009
Messages
95
Just 4 static green LEDs on the SAS Expander card? I have the same with my yellow PCB and a HP P410 array controller.
 

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
Yes, just static green LEDs. I can't see how many due to it's location in the case, but I'd say at least 3 are on and it could very well be 4.
 

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
Just as an update on my testing:
SAS Expander PCB Color: Green
Manufacture Date: Nov 2009
Firmware Version: 1.00

Raid Cards tested: (results)
1. Adaptec 5805: Expander Not Recognized
2. Adaptec 31605: Expander Not Recognized
3. Adaptec 3085: Expander Not Recognized

Raid Cards left to test:
1. Areca ARC-1680LP
2. Dell Perc 5/i

Drives Attached to Expander:
1. Seagate 7200.11 1TB
2. Seagate 7200.11 1.5TB
3. Hitachi 7200rpm 1TB
4. Hitachi 7200rpm 2TB
5. Western Digital Green 1.5TB
 

novadude

[H]ard|Gawd
Joined
Jul 8, 2004
Messages
1,108
Just as an update on my testing:
SAS Expander PCB Color: Green
Manufacture Date: Nov 2009
Firmware Version: 1.00

Raid Cards tested: (results)
1. Adaptec 5805: Expander Not Recognized
2. Adaptec 31605: Expander Not Recognized
3. Adaptec 3085: Expander Not Recognized

Raid Cards left to test:
1. Areca ARC-1680LP
2. Dell Perc 5/i

Drives Attached to Expander:
1. Seagate 7200.11 1TB
2. Seagate 7200.11 1.5TB
3. Hitachi 7200rpm 1TB
4. Hitachi 7200rpm 2TB
5. Western Digital Green 1.5TB

Update the expander's firmware to 1.52 or later and it'll work.
 

BENN0

Weaksauce
Joined
Nov 23, 2009
Messages
95
Update the expander's firmware to 1.52 or later and it'll work.

How? I tried this with the latest HP firware update CD (on a bootable thumb drive) and while the update software recognises my HP P410 controller it never sees the Expander Card to update.
 

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
why dont you guys try this?

http://usa.chenbro.com/corporatesite/products_detail.php?sku=74 goes for $230 if you search for it on google product search and its not off of ebay.

i think its compatible...

Because the Chenbro's RARELY show up on ebay and they're extremely hard to get anywhere else online - has to be special ordered direct from manufacturer if you do find it online somewhere, reason being they're discontinued as Chenbro preps a new SAS2 line for market.

The whole point of this thread is people looking for an alternative to the Chenbro stuff, something that also has a bit better compatibility all around due to the PMC chip on the HP > LSI chip on existing Chenbro's.
 

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
Waiting for my P410 at this point. Seems like once the HP SAS expander gets up and running, it is solid, whereas the partial workings of the Chenbro one make me a bit more scared. Also, the HP one is < $200 for the green PCB version.
 

myrison

Weaksauce
Joined
Dec 16, 2009
Messages
116
Where can you get it for that price?

It shows up pretty regularly on eBay. Obviously from reading this thread you have to be careful which version you buy and from whom, but I've been watching ebay for a couple of months for the card and there is almost always at least one there for ~200 and sometimes less.
 

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
I've bought about 4 of these cards from a particular seller I found through ebay, I won't mention name due to forum selling rules but you can PM me for contact info. I've been getting them for about $150 ea. The guy sells on ebay for $190 but if you buy direct he'll do it for $150-$175. Last I asked he had about 20 on hand.

However the firmware that ships on those green PCB cards is v1.00, and you'll need 1.52 for proper 3Gbps operation with newer SATA drives. The latest HP SAS expander firmware is now 2.02, but it didn't add anything requisite- just some sort of update due to a new hardware rev. I updated my four cards from 1.00 to 2.02 and they've been solid.

Today I'm testing how many drives max I can attach - it looks possible to fill up the 8 miniSAS connectors with 32 drives and attaching to my Areca 1680 via SFF-8088 connector, since the expander really has no discrete "input ports" -- HP documentation designates connectors #8 and #9 as 'input' but its trivial- there's nothing unique about those ports versus any other port. This card and SAS expanders in general can be thought of like an ethernet switch in functionality- any port works for input/output at least in my testing. I'll be posting these and more results in Post #1 when I'm all done, including daisy chaining, how to flash the firmware properly using an undocumented workaround, etc.
 
Last edited:

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
This is interesting - 8 x Seagate 1.5Tb CC1H drives perform flawless in RAID6 when connected to an HP P411 + HP SAS expander. If people remember these are the 'dreaded' Seagates that gave everyone troubles. These drives always got kicked out of the array in under an hour when I connected them to an Areca 1680ix-24 or Adaptec 52445 due to what the manufacturers claimed was a bug in the IOP348 code.

I've sometimes wondered if some of the 'incompatibilities' between certain drives and high port count IOP348-based cards weren't just an issue of the onboard expander on these cards rather than the raid chip-- I'm guessing that using the HP expander with a low port count Areca or Adaptec might just open up more drive compatibility.
 

novadude

[H]ard|Gawd
Joined
Jul 8, 2004
Messages
1,108
I've sometimes wondered if some of the 'incompatibilities' between certain drives and high port count IOP348-based cards weren't just an issue of the onboard expander on these cards rather than the raid chip-- I'm guessing that using the HP expander with a low port count Areca or Adaptec might just open up more drive compatibility.

Time to start checking all of the drives against those low port cards and get a list going then.
Off the top of my head the 2tb seagate LP's work fine on a bare 5805, the WD GP's should work too, as do the Hitachi's. That leaves the samsung f3eg again.

None of these drives have TLER equivalents enabled so I wouldn't trust them to high drive number arrays.
 

BENN0

Weaksauce
Joined
Nov 23, 2009
Messages
95
If there is anyone using this card with a HP P410 controller and you are still experimenting, could you please check if there is a logical drive limit?
The largest logical drive I created was 12TB (6x 2TB drives in RAID0) but that should be about the limit of the P410 card according to HP (12TB or 14TB, you'll find both mentioned on the HP site).
I'm hoping this is a physical limit of the HP servers and not the real limit of the P410 controller.
 

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
Lulz -- just finished testing a 32-drive Raid6 array (temporary- not sure I'd want to run 32-drive array longterm), so obviously connecting drives to all 8 of the SFF-8087 connectors is not a problem. Connected the HP SAS Expander to Areca 1680 via SFF-8088 cable on the external port. Screenshots to follow.

Also I highly recommend a 40x40x10mm fan for this card, not to mention any raid card that has a heatsink but shipped without a fan. Reason being the HP SAS Expander was designed with high CFM airflow HP server cases in mind, and so lower airflow cases like a Norco 4220 might allow the expander chip to heat up, and when it gets too hot I notice I/O throughput drop a lot lower on benchmarks, and on one occasion when it got too hot, some drives weren't being seen by the array controller anymore- that situation was my fault since I had placed it between two other cards with no airflow.

I've bought several of these fans and stuck them on my expanders, Areca and Adaptec cards that didn't already have a fan. Comes with screws that let you attach fan directly to heatsink.
http://www.newegg.com/Product/Product.aspx?Item=N82E16835119049
 
Last edited:

alamone

Gawd
Joined
Sep 30, 2008
Messages
589
In my case I just use a 120mm fan ghetto-modded (velcro) perpendicular to the PCIe slots to cool all of my cards. The 40mm fan on my Highpoint 4320 started making grinding noises so I just ripped it off and let the 120mm fan cool it off. The Areca 1231 also gets quite hot unless you cool it off actively. While not as effective as a direct fan on the heatsinks, it works well enough.

Also, I never had overheating problems w/ my Chenbro SAS expander which is only passively sinked, but nothing else is in the case as it's just a JBOD. I might stick a PCI cooling fan since I have one lying around, though.
 

SeanG

n00b
Joined
Oct 13, 2005
Messages
43
However the firmware that ships on those green PCB cards is v1.00, and you'll need 1.52 for proper 3Gbps operation with newer SATA drives. The latest HP SAS expander firmware is now 2.02, but it didn't add anything requisite- just some sort of update due to a new hardware rev. I updated my four cards from 1.00 to 2.02 and they've been solid.

Were you able to flash the cards to 2.02 without other HP hardware or did you have to use an HP server and P410 controller?
 

nitrobass24

[H]ard|DCer of the Month - December 2009
Joined
Apr 7, 2006
Messages
10,462
Were you able to flash the cards to 2.02 without other HP hardware or did you have to use an HP server and P410 controller?

Talked to odditory last night and he told me you have to use the HP controller.
 

SeanG

n00b
Joined
Oct 13, 2005
Messages
43
Talked to odditory last night and he told me you have to use the HP controller.

Boo... but figures it would be that way. Did he have to use an HP server as well or a home built is ok?

If an HP server is required, I'll have to take mine in to work and look for an appropriate machine. Might need a flashing service for these cards!
 

BENN0

Weaksauce
Joined
Nov 23, 2009
Messages
95
You don't need a HP server but you'll have to circumvent the check on the firmware update CD that tests for HP hardware. I created a bootable USB pen drive from the Firmware Update CD (as described on the HP site). In the root of the USB pen drive there is a file 'blacklist'. I edited out all the values between the "<ids> </ ids>" tags (the file is read-only so change its attributes first). Now the firmware update utility will run without checking if is running on supported HP hardware.

If you just burned the ISO to a CD you can also boot that and if you hold left CTRL key and then press enter when the first menu comes up "HP Firmware Maintenance 8.60" then it won't give you the stop message that it's not an HP system. (thanks odditory for this work around).
 

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
EDIT: A lot of updates to the OP (Post #1). Will add firmware flashing step-by-step tomorrow. As BENN0 mentioned, yes there's a CTRL-ENTER workaround to bypass the HP hardware check but it still requires an HP raid card.
 
Last edited:

SeanG

n00b
Joined
Oct 13, 2005
Messages
43
FYI, the 3ware 9690SA cards works with the SAS Expander quite well. In fact, my expander has an older firmware and the Areca 1680ix-12 I have hangs when I connect it via the external port. The 3ware does not. The 3ware only does SATA-I (1.5Gbps) however. I'm going to try getting it flashed soon to see if this fixes the hanging issue.

From a performance standpoint, the Areca beats the 3ware hands down. Plus the Areca supports the Advanced Power Management features which is key in keeping electrical costs down in a home environment.
 

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
@SeanG: thanks for 3Ware info, I'll add it to the HCL and reference you. You're right about Areca performance, and where they really outrun everything else is in array build / rebuild / verify times. 19 hours to build a 20 x 2Tb Raid6 = WIN.

How are you connecting your Areca 1680ix-12 to the HP expander? You should be using the external SFF-8088 connector. If you tried using the internal SFF-8087 ports to connect to the HP that could be your issue, since you're connecting an expander to another expander essentially- if you notice the heatsink closest to the SFF-8087 connectors, that's your onboard expander, hence the "ix" a.k.a. internal expander designation in the product name. That SFF-8088 connector OTOH doesn't route through the onboard expander and talks straight to the raid chip. Daisy chaining expanders is *supposed* to work as far as SAS specs go, but the IX on the Areca and Adaptec cards don't play nicely with a lot of stuff.
 
Last edited:

BENN0

Weaksauce
Joined
Nov 23, 2009
Messages
95
From the first post:

HP P212, P410, P410i, P411, P411i, P712: YES (However I wouldn't recommend these controllers however from a performance standpoint)

Did you do performance benchmarks or are you talking about RAID creation/initialization performance?
I was able to get transfer speeds that seemed like the PCIe bus limit in some benchmarks (RAID0 sequential) and am seeing very acceptable performance in real life (home use) situations as well with the P410 (~180MB/sec read and write throughput on RAID5, about what two aggregated Gigabit Ethernet connections can handle).
 
Last edited:

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
No dice on the 5805? I have a second v1.0 expander + a HP P410/512MB + BBU at a FedEx location about 15 miles away (delivery scheduled for tomorrow).

Right now in the main rig I have (all running):
Areca 1680LP
Adaptec 3085
Adaptec 31605
Adaptec 5805

Add the P410 in there tomorrow and that will be 5 cards + 2 SAS expanders I can play with all in one chassis :)

And BTW, great post!
 

SeanG

n00b
Joined
Oct 13, 2005
Messages
43
@SeanG: thanks for 3Ware info, I'll add it to the HCL and reference you. You're right about Areca performance, and where they really outrun everything else is in array build / rebuild / verify times. 19 hours to build a 20 x 2Tb Raid6 = WIN.

How are you connecting your Areca 1680ix-12 to the HP expander? You should be using the external SFF-8088 connector. If you tried using the internal SFF-8087 ports to connect to the HP that could be your issue, since you're connecting an expander to another expander essentially- if you notice the heatsink closest to the SFF-8087 connectors, that's your onboard expander, hence the "ix" a.k.a. internal expander designation in the product name. That SFF-8088 connector OTOH doesn't route through the onboard expander and talks straight to the raid chip. Daisy chaining expanders is *supposed* to work as far as SAS specs go, but the IX on the Areca and Adaptec cards don't play nicely with a lot of stuff.

If I use the external connector on the 1680ix-12 to the external connector on the SAS expander, the Areca hangs during post and eventually reboots due to a timeout. If I connect the 1680ix-12 internal port (any one) to the internal port on the SAS expander, it works fine. I don't have any SFF-8087 to SFF-8088 cables so I can't try internal to external either way.

The 3ware 9690SA I have has external ports only. External to external works fine but the 3ware controller reports that it's operating at 1.5Gbps (SATA I).

The only two issues I can think of is that I don't have the latest firmware on the HP SAS expander so I'll need to get my hands on a Pxxx HP RAID controller to flash it. The other issue is that you can configure the cable length on the Areca thru the serial port. By default it's set to 1m for internal and 3m for external. My external cables are short, 0.5m and 1m so it could be a signaling issue. Problem is that I followed the instruction to set the cable length for the Areca's external port but it doesn't stick after a reboot. Performance via the internal to internal cable was excellent - almost three times as fast as the 3ware controller (8 Hitachis in RAID 5).

For my implementation, I will probably only use 3 Norco 4224 cases cuz I'm limited to 12U in my rack (when Norco finally releases them) which is why I got the -12 card. That's three ports. I can easily cable that to the outside of the machine using one of those SFF-8087 to SFF-8088 backplates.
 

nitrobass24

[H]ard|DCer of the Month - December 2009
Joined
Apr 7, 2006
Messages
10,462
SeanG

Does your Areca have the latest firmware?
Cable lenght should stick after a reboot, so that makes me think there is something off with your card.
 

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
Sean if your HP expander is running less than firmware 1.52 then its going to be stuck at 1.5Gbps until you upgrade. Check your PM.
 

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
Which version of the Firmware Maintenance CD are you using? 8.70? I have a P410 w/ 2.72 firmware that can't see the V1.00 expanders.
 
Last edited:

odditory

Supreme [H]ardness
Joined
Dec 23, 2007
Messages
6,482
I guess I better make that firmware walkthrough in the OP.

roughly: I think I used 8.60 but doesn't matter, you need to use the HP USB flash drive utility which will convert the cd image to bootable usb, after that's done go and download firmware 2.02 supplemental file (it has extension .scexe) and copy it to the flash drive .. One of the folders contains lots of other .scexe files .. Look for it and drop it in there. Then boot with usb drive, when first prompt comes up hold left-ctrl and press enter and it should say Booting... Then follow prompts.

I've updated four 1.0 cards to 2.02 with no probs. I did say a little prayer to jesus beforehand though so that may have factored, even though technically I cursed him out for something totally unrelated.
 
Last edited:

dobermanUK

n00b
Joined
Jan 21, 2010
Messages
1
I have been following the thread since the beginning, and yeah, this is my first post!
I like the idea of the numerous drives but how reliable is a raid configuration? If something goes wrong, everything is gonne correct? How often do things go wrong?
I was thinking of building a media server with the possibility of expansion, and this thread gave me great ideas on how things work.
 

Neme

n00b
Joined
Jan 19, 2010
Messages
30
I guess I better make that firmware walkthrough in the OP.

roughly: I think I used 8.60 but doesn't matter, you need to use the HP USB flash drive utility which will convert the cd image to bootable usb, after that's done go and download firmware 2.02 supplemental file (it has extension .scexe) and copy it to the flash drive .. One of the folders contains lots of other .scexe files .. Look for it and drop it in there. Then boot with usb drive, when first prompt comes up hold left-ctrl and press enter and it should say Booting... Then follow prompts.

I've updated four 1.0 cards to 2.02 with no probs. I did say a little prayer to jesus beforehand though so that may have factored, even though technically I cursed him out for something totally unrelated.

I presume even with bypassing the HP systems check (leftCTRL+Enter) you still need one of the supported HP raid boards? (P212, P410, P411, or P712) to flash the expander?
 

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
I guess I better make that firmware walkthrough in the OP.

roughly: I think I used 8.60 but doesn't matter, you need to use the HP USB flash drive utility which will convert the cd image to bootable usb, after that's done go and download firmware 2.02 supplemental file (it has extension .scexe) and copy it to the flash drive .. One of the folders contains lots of other .scexe files .. Look for it and drop it in there. Then boot with usb drive, when first prompt comes up hold left-ctrl and press enter and it should say Booting... Then follow prompts.

I've updated four 1.0 cards to 2.02 with no probs. I did say a little prayer to jesus beforehand though so that may have factored, even though technically I cursed him out for something totally unrelated.

OK will give that a shot. I've been using the boot CD which hasn't been detecting the expander. I saw a way in HP SUM to add additional checks by plugging in different media so I'll give that a shot then try the USB method.
 
Top