HP SAS Expander Owner's Thread

nitrobass24 said:
Yea i have red LEDs.

Also if you are designing your own Backplane it would be cheaper to develop some that dont have onboard logic, like those in the Norco's. This would also make your activity/fail LED the same, but it would be a simpler(read: Cheaper) design and the end result is nearly the same, and would provide adequate information to a user
Click to expand...

I was thinking of incorporating the expander directly on the backplane so that I don't have to mess around with cables. I looked at a few different expander chips but so far I like the PM8005 the best, especially all of the additional functionality that PMC-Sierra build into it. The setup I have in mind would require two of them. The power, activity and fault LEDs would be on a separate board.
 
@treadstone, FYI: Norco is looking for a partner to help design a SAS expander backplane for their server cases. I don't know if its something up your alley - the idea of designing a board for them, but I can put you in touch if interested. They've been trying to design their own board and having some issues- I suspect they haven't found an ideal SAS chip but I'm going to ask them again.
 
Last edited:
I would like to share my experience with daisy-chaining the expanders. My setup looks like this:
Code: 
ARC-1680x <----Expander1----> Port 1                     Port 1 Not Used
                              Port 2 Not Used            Port 2 Drive 01-04
                              Port 3 Drive 05-08         Port 3 Not Used
                              Port 4 Drive 09-12         Port 4 Not Used
                              Port 5 Drive 13-16         Port 5 Not Used
                              Port 6 Drive 17-20         Port 6 Not Used
                              Port 7 Not Used            Port 7 Not Used
                              Port 8 Not Used            Port 8 Not Used
                              Port 9 <----Expander2----> Port 9

I only have one RPC-4220 populated with 15 drives to work with. I moved the drives on Port 2 (Expander1) to Port 2 (Expander2) for testing.
Both expanders are inter-connected on Port 9.

I can see both expanders and drives in web interface. I can also use the drives on Port 2 (Expander2) in my OS. I believe this to be working. :)

Screenshot:
http://inode.se/img/hpchain/arecasas.png
http://inode.se/img/hpchain/hpsas1.png
http://inode.se/img/hpchain/hpsas2.png

Firmware:
Areca ARC-1680x, v1.48
HP SAS Expander, v2.02

EDIT1: I noticed Port 9 is missing on the web interface, but the rest is listed.
EDIT2: Weird, Port 2 is listed as Port 1 on Expander2.
 
Last edited:
Nice, thanks for the confirmation, fyxim.

I've had a similar experience but wanted to do more testing before I updated the OP. I wasn't sure if there were only specific ports that allowed the daisy chaining or it was any port -- so far it seems like any port can be used, but I still need to test if chaining two expanders with the external SFF-8088 port is possible, when using an internal connection from the first expander to the host card.
 
Thanks for posting about daisy-chaining the HP Expanders. I was wondering about that.

@treadstone: any idea what is different in your system that daisy-chaining failed to work?
 
@ john4200: not sure why, but when I connected them in a daisy chain setup it did not work for me.

@ fyxim: Just to confirm (so that we are all on the same page):

This is the official (HP designated) port numbering:

Port 1: External port
Port 2: Port 2C - To Backplane - Internal - top edge of the board
Port 3: Port 3C - To Backplane - Internal - top edge of the board
Port 4: Port 4C - To Backplane - Internal
Port 5: Port 5C - To Backplane - Internal
Port 6: Port 6C - To Backplane - Internal
Port 7: Port 7C - To Backplane - Internal
Port 8: Port 8C - To Controller - Internal - bottom edge of the board
Port 9: Port 9C - To Controller - Internal - bottom edge of the board

The ports that are connected to the controller usually do not show up in the Areca web interface!
 
odditory said:
@treadstone, FYI: Norco is looking for a partner to help design a SAS expander backplane for their server cases. I don't know if its something up your alley - the idea of designing a board for them, but I can put you in touch if interested. They've been trying to design their own board and having some issues- I suspect they haven't found an ideal SAS chip but I'm going to ask them again.
Click to expand...

I might consider it, but there are a few things that may cause some issues (legal things that need to be addressed first since the information from PMC-Sierra are under NDA)!

I have to think about it...
 
treadstone said:
Port 1: External port
Port 2: Port 2C - To Backplane - Internal - top edge of the board
Port 3: Port 3C - To Backplane - Internal - top edge of the board
Port 4: Port 4C - To Backplane - Internal
Port 5: Port 5C - To Backplane - Internal
Port 6: Port 6C - To Backplane - Internal
Port 7: Port 7C - To Backplane - Internal
Port 8: Port 8C - To Controller - Internal - bottom edge of the board
Port 9: Port 9C - To Controller - Internal - bottom edge of the board
Click to expand...
Correct, I actually modified your code from a earlier post.
 
fyxim said:
Correct, I actually modified your code from a earlier post.
Click to expand...

Cool :)

So you use a SFF8088 to SFF8088 cable to connect the controller to the first HP Expander (externally) and then a SFF8087 to SFF8087 cable from port 9C to port 9c in between the two HP Expanders (internally), correct?
 
Dude, all the activity in this thread is getting me excited! My expander and most of my cables will be here by Friday. The setup I intend to run will be the HP SAS expander connected to a LSI 1068E controller that's integrated into my Asus P5BV-SAS mobo. I also have a 3ware 9650SE-4LPME controller that I can test the expander with.

Also, this is a document that I have found helpful regarding cable questions. I'm betting most of y'all don't need it or already have seen it, but I thought I'd post it anyway.

http://www.3ware.com/products/pdf/3ware_Cable_Brochure.pdf
 
Will a 3ware 9650 work with a SAS Expander? I'm not familiar with the 3ware "sidecar" or what it does.
 
Not sure, but I'll give it a try.

Edit: It doesn't look like the 9650SE supports expanders, but I also can seem to find where it says it doesn't either.
 
Last edited:
treadstone said:
Cool :)

So you use a SFF8088 to SFF8088 cable to connect the controller to the first HP Expander (externally) and then a SFF8087 to SFF8087 cable from port 9C to port 9c in between the two HP Expanders (internally), correct?
Click to expand...
Yes, I'm not sure if the SFF8087 cables I'm using have sideband or not, but I use same cables for both expanders and backplane.
 
I thought the 9650 was a SATA controller. I think you need a SAS-aware controller to use with a SAS Expander. But I do not know for certain.

I've used the 3ware 9690SA with a SAS expander EL2 backplane in a Supermicro enclosure. It kind of worked, but had some odd problems (possibly because it was EL2 being used with 1 controller, where I should have had an EL1).
 
john4200 said:
I thought the 9650 was a SATA controller. I think you need a SAS-aware controller to use with a SAS Expander. But I do not know for certain.

I've used the 3ware 9690SA with a SAS expander EL2 backplane in a Supermicro enclosure. It kind of worked, but had some odd problems (possibly because it was EL2 being used with 1 controller, where I should have had an EL1).
Click to expand...

I'm pretty sure you're right, but I'm going to try it anyway, not going to hurt anything.
 
Ok, I got my 1680LP in about two hours ago. Tossed it in the system with the expander turned off SES2, and it works. I love when things just work like they should.

So my issue with the 3ware 9690SA was that it doesn't support the newer firmware on the sas expander.
 
I have questions regarding my setup, so if someone can help me before I take an plunge as Im upgrading mine Core2Duo/775 server;

1. I will be using SASLP-MV card with this expander. If I would theoretically connect all 32 hard drives to the extender, will SASLP be able to handle all data rates from drives without throttling speeds? The x4 pcie ports is operating with 16 Gbps (2 GB/sec) so if all drives will be operating it will throttle them down.. If so, am I better with 4 separate SASLP cards (motherboard has enough pcie slots for separate cards)

2. Would drives connected thru saslp-mv8 + expander pass the SMART info to OS?
 
Last edited:
JonTa said:
1. I will be using SASLP-MV card with this expander. If I would theoretically connect all 32 hard drives to the extender, will SASLP be able to handle all data rates from drives without throttling speeds? The x4 pcie ports is operating with 16 Gbps (2 GB/sec) so if all drives will be operating it will throttle them down.. If so, am I better with 4 separate SASLP cards (motherboard has enough pcie slots for separate cards)
Click to expand...

What are you doing that needs >2GB/sec? Is it really going to be an issue?

Note that even in RAID0 or RAID5/6 you are unlikely to ever see speeds greater than that from a single array anyway, and you won't bottleneck the drives for anything but pure sequential performance.
 
If you really need those kinds of speeds you are better off with a pair of LSI 9211-8i cards.
 
JonTa said:
The x4 pcie ports is operating with 16 Gbps (2 GB/sec) so if all drives will be operating it will throttle them down.. If so, am I better with 4 separate SASLP cards (motherboard has enough pcie slots for separate cards)
Click to expand...

What motherboard (more specifically, north bridge) are you using? You need to know how many dedicated PCI-E lanes are available and if they are 1.0 (250 MB/sec/lane) or 2.0 (500 MB/sec/lane). A lot of consumer Intel motherboards with more than 2-3 PCI-E slots use the nVidia NF200 PCI-E bridge (more like switch) which divides up (switches) dedicated PCI-E lanes among multiple so-called "x16" slots.

Second, you need to find out if the Marvell 6480 controller on the MVSAS card uses native SATA or SATA-over-SAS (STP) to talk to the drives. If it's the latter you'll be limited to about 700 MB/sec regardless of the number of hard drives.

If you want just one array, I think you'd probably need *SAS* drives along with a SAS3 RAID controller with cache (like the LSI 9260 series) to get anywhere near sustained 2GB/sec throughput.

Edit: or as nitrobass suggested, multiple SAS3 controllers *without* an expander, and probably software-stripe the (RAID) arrays together for speed.
 
Just wondering if there's a decimal point off in this:

Q: How many cables do I need to connect the HP SAS expander to my RAID or HBA card?
A: You only need 1, at 2.4Gbps it's enough bandwidth for all your drives, however 2 cables provide a 4.8Gbps interlink. In benchmark tests I didn't see any performance difference between using 1 and 2 interlink cables. That may change however when the expander is combined with SAS2 RAID cards and/or SATA3 harddisks. When a SAS2 RAID or non-RAID HBA drives this expander, the interlink speeds are double (4.8Gbps for single cable, 9.6Gbps for two cables).
 
Should be GB(ytes)ps, not Gbps. It's the full-duplex bandwidth. 1 SAS lane = 1 SATA2 link = 300 MBps; one multi-lane cable gives you 4 lanes = 1200 MBps. Full-duplexed that's 2400 MBps = 2.4 GBps.

SAS2 (& SATA3) double per-lane rate to 600 MBps.
 
Sounds right to me. With 8b/10b encoding, 3.0 Gbps SAS turns into 2.4 Gbps effective bandwidth.

Although I was under the assumption that the 3.0 Gbps connection was on a per lane basis, with the one cable carrying 4 lanes, so 4x3.0Gbps for SAS and 4x6.0 for SAS2 per cable?
 
izx said:
Should be GB(ytes)ps, not Gbps. It's the full-duplex bandwidth. 1 SAS lane = 1 SATA2 link = 300 MBps; one multi-lane cable gives you 4 lanes = 1200 MBps. Full-duplexed that's 2400 MBps = 2.4 GBps.

SAS2 (& SATA3) double per-lane rate to 600 MBps.
Click to expand...

So I take it that the reported SATA and SAS rates are given in half duplex? I plan on using a SAS2 card so I can expect 2400MBps half duplex and 4800MBps full duplex.

So I guess then that one multilane connector can easily drive 20-24 regular SATA mechanical drives at their full speed simultaneously. Impressive. I should have no problems at all for a home server.
 
Yes, but there are certain caveats if you use *SATA* drives with a SAS controller/expander; SATA commands/packets are encapsulated within SAS using the "Serial Tunneling Protocol" which limits throughput over a 4-lane connector to ~700 MBps instead of the expected 1200; for SAS2 this doubles to ~1400 MBps. I'm not sure if you can circumvent this limitation by using a dual-cable interlink...we'll have to wait for odditory to chime in.
 
I still get 800mb/s sustained and 1063mb/s burst with my 1680i and SAS expander.
 
izx said:
Yes, but there are certain caveats if you use *SATA* drives with a SAS controller/expander; SATA commands/packets are encapsulated within SAS using the "Serial Tunneling Protocol" which limits throughput over a 4-lane connector to ~700 MBps instead of the expected 1200; for SAS2 this doubles to ~1400 MBps. I'm not sure if you can circumvent this limitation by using a dual-cable interlink...we'll have to wait for odditory to chime in.
Click to expand...

I think that is more than sufficient since my ultimate bottleneck is really 1000mbps ethernet. I doubt that it even slow down WHS drive extender rebalancing. It seems it is faster than even having SATA drives connected to individual SATA ports.

EDIT: Now I get what you're saying. My whole array would be limited by that speed. Yes, I would like to know if the dual cable interlink would solve that issue.
 
Last edited:
sxr71 said:
So I take it that the reported SATA and SAS rates are given in half duplex? I plan on using a SAS2 card so I can expect 2400MBps half duplex and 4800MBps full duplex.

So I guess then that one multilane connector can easily drive 20-24 regular SATA mechanical drives at their full speed simultaneously. Impressive. I should have no problems at all for a home server.
Click to expand...

SATA rates are always reported in half duplex. For the simple reason that SATA is half duplex. SAS on the other hand is full duplex.

If you are using SATA drives then the full-duplex SAS capability is not really of use unless you have separate arrays on a single SAS card. An array of SATA drives can only do reads OR writes at once, so if you only have a single array of SATA drives your SAS card/expander effectively become half duplex also. If you do have 2 or more arrays then 1 could be performing a read operation while the other is writing however, and you will potentially use more of the available SAS bandwidth.
 
sxr71 said:
I think that is more than sufficient since my ultimate bottleneck is really 1000mbps ethernet.
Click to expand...

OK, so if you're limited by GigE to a max of 125 Mbytes/sec, hell, we'll say 500 Mbytes/sec theoretical if you get 4 NICs and do link aggregation (teaming), you're still well below the 700-800 Mbytes/sec a single cable interlink will give you. Why worry?
 
the_beast said:
SATA rates are always reported in half duplex. For the simple reason that SATA is half duplex. SAS on the other hand is full duplex.

If you are using SATA drives then the full-duplex SAS capability is not really of use unless you have separate arrays on a single SAS card. An array of SATA drives can only do reads OR writes at once, so if you only have a single array of SATA drives your SAS card/expander effectively become half duplex also. If you do have 2 or more arrays then 1 could be performing a read operation while the other is writing however, and you will potentially use more of the available SAS bandwidth.
Click to expand...

So those Seagate 7200RPM SAS drives do have an advantage over their SATA relatives. :eek:
 
Krobar said:
So those Seagate 7200RPM SAS drives do have an advantage over their SATA relatives. :eek:
Click to expand...

SAS has (technically) 4 times the bandwidth of SATA, as it is full duplex and each drive has 2 ports. Problem is you can't use both ports for data transfer at the same time. One can be used for commands and the other data transfer simultaneously though. But realistically under the kind of loading 'normal' home users will see the 2 will perform very similarly, because the command overhead is not all that great anyway. The full duplex nature isn't really all that useful for a home user either.

Most people would be better off buying multiple SATA drives rather than a single 7200rpm SAS disk for the same money.
 
the 2 ports on SAS is for multipath, that's fault tolerance feature.
"Should one controller, port or switch fail, the operating system can route I/O through the remaining controller transparently to the application, with no changes visible to the applications, other than perhaps incremental latency."
Click to expand...
 
Sorry if I missed this, I have a 1068E HBA, I also have both the Chenbro expander (rev B0) and the HP expander. Both work exactly the same for me, except that the Chenbro is much easier to power (no pci-e slot needed). Whats the point in trying to hack something up to power a HP Expander w/o a motherboard when Chenbro Expander is available?

I've seen people mention the Chenbro is difficult to find, was that not the problem with the first version (rev A0)? I ordered a rev B0 Chenbro expander with no effort.
 
IIRC, some folks with Areca RAID controllers had issues with the Chenbro expander. Also, the HP is SAS2 and somewhat futureproof. From the little bit of research I did, the HPs are (were?) also about ~$50 or so cheaper if purchased from Dustin.
 
Hmm, were the problems with the older A0 rev or both the A0 and B0 revs? I thought the B0 had a newer chip to fix problems. SAS2 is nice. Prices are the same. I got the Chenbro for a few dollars less than the HP. $229 vs $234. Oh yeah I got the CK12804, which has no external connector-- that another downside. Still, seems easier to ghetto rig an external -> internal minisas connector than something to provide power via pcie.
 
I just picked up my backplane for the SAS expander. Looks pretty good. Just have to order another expander and an adapter and I'll be set. 40 hot-swap bays should keep me occupied for a while.
 
bexamous said:
Hmm, were the problems with the older A0 rev or both the A0 and B0 revs? I thought the B0 had a newer chip to fix problems. SAS2 is nice. Prices are the same. I got the Chenbro for a few dollars less than the HP. $229 vs $234. Oh yeah I got the CK12804, which has no external connector-- that another downside. Still, seems easier to ghetto rig an external -> internal minisas connector than something to provide power via pcie.
Click to expand...

Not sure about the different revs...I just wanted something that works. As for PCI-E power, a number of robust solutions are discussed earlier in this thread. An 8087->8088 PCI slot adapter is about $29, so not really much cheaper so long as you still have a power supply in the chassis with the hard drives.
 
bexamous said:
Hmm, were the problems with the older A0 rev or both the A0 and B0 revs? I thought the B0 had a newer chip to fix problems. SAS2 is nice. Prices are the same. I got the Chenbro for a few dollars less than the HP. $229 vs $234. Oh yeah I got the CK12804, which has no external connector-- that another downside. Still, seems easier to ghetto rig an external -> internal minisas connector than something to provide power via pcie.
Click to expand...

I just googled for Chenbro SAS expanders, and although a number of web stores list them, I did not find any that indicated they were in stock.

Where did you order yours from?
 
The only place I've seen them repeatedly are eBay. There are 2-3 new listings per week for the expanders, they usually go for ~300 or a bit less depending on the exact model.
 
You must log in or register to reply here.
Back
Top