JBOD woes

Kurlon

Weaksauce
Joined
Dec 1, 2004
Messages
72
So, I decided to build a big old OI based storage mule. I started with a HP DL380G6, dual cpus, 80GB RAM, and the onboard RAID driving it's 2.5" SAS cluster as a RAID 5 for the OS. Not ideal but the HP RAID controller doesn't do IT mode. I may break it out into dual RAID 5s and let OI mirror them for extra piece of mind...

For storage control I got a LSI 9207-8e. This would feed an Addonics RR2035RSDES shelf with 20 bays. To keep things snappy I also sourced three Intel DC3600 400GB PCIe SSDs, two to be mirrored as a log volume, one for cache. I've got an existing fiberchanel fed farm that I want to tie into so I've tossed in a dual 4GB port FC card. From there it'd be a happy time just filling it up with disks. I opted to start with five WD Red 6TB units as a test.

My research showed the LSI dealt with multiple disks off a given port, and I knew SAS controllers would talk to SATA so I thought all was good. I failed to discover that 'SATA port multipliers' and 'SAS port multipliers' are not treated equally. Net result at most I could get four drives recognized, one off each SATA PM in the chassis. Research fail on my part.

I'm now stuck trying to massage this collection of bits back into usable service for as low a cost as possible. As the Addonics case is JUST a drive enclosure I'm limited to a SAS expander that can be powered by a molex connector or similar, there isn't a PCIe slot in there. That naturally has led me to the Chenbro CK23601. Ideally I'd like to go SFF-8088 to SFF-8088 from my LSI to the CK, possibly dual linking. I'll be able to mount the CK using a plexi-adapter sheet in the case to accommodate this nicely. The docs and Chenbro support say no, they insist you must use the one designated SFF-8087 port as your input from the HBA. I've seen user reports that claim otherwise in this forum, but nothing recent.

Anyone care to post a current update on the CK and how flexible it is on port designation?

I've also got to chase why OI only sees two out of three NVMe SSDs but that's a problem I can solve down the road.
 
Why don't you use a HP SAS expander and one of these? You can ignore the USB cable and PCIe x1 part. They're not needed. You can either get a SFF-8088 to SFF-8087 cable or get a PCI bracket that does the conversion.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
A quick look shows the HP expanders costing easily twice as much as the Chenbro new. That puts me close to the 'scrap the project, return it all' price range. (eBay/used/Amazon generally isn't usually a viable option for me in this scenario.) What's the HP unit bring to the table in comparison to the Chenbro?
 
I don't know if it brings anything. I would never recommend buying the HP at full price. At $65 it's killer. Your post makes it sound like cobbling together something at home in your basement, but you can't buy from eBay / used / Amazon? So you're cobbling something together for an enterprise environment? :confused:
 
Last edited:
It's the worst of both, 'cobbled together' for a work env. I'm reusing a bunch of bits that have been decommissioned, combining with some new hardware to get extended use out of it. Because I'm doing it through work I have to go through a purchasing process that makes using eBay/Amazon/etc much more difficult.
 
You don't have a purchasing credit card you can use to buy stuff? Or you can't buy it yourself and expense it? The company would rather pay more? The insanity of companies never ceases to amaze me.
 
Both company CCs and expenses have been abused in the past, so they're locked down a bit. That's what I mean by eBay/Amazon/etc being much more difficult. I can do it, but there is a time value equation to be factored in. If the Chenbro and HP unit are functionally equivalent, I may do the leg work to go the HP path. If the Chenbro nets me anything over the HP... I'm already approved to buy it.

At the moment, unless I'm misreading something it looks like the HP expander only does 3Gb/s SATA while the Chenbro (and Intel based on the same chip) do 6Gb/s SATA?
 
If you're buying an LSI SAS2x36-based expander brand new in the US, you get the Intel RES2CV360, not Chenbro (same price). Yes, you'll need a dumb dual-port 8088-8087 adapter ($30), but you get Intel-quality with an Intel-warranty.

As for dual-linking or which port to connect to the initiator (HBA), Chenbro is full of horseshit. From the Intel hardware guide for the 6-port SAS2x24 model:

  • Any phys may be combined into wide port(s)
  • Allows any number of phys to be included in a wide port
  • PHY[3:0] supports optical links between LSI Gen2 SAS device

"Dual-linking" is technically termed an x8 wide port. So there, no limitations on the Intel version unless you're using fiber links. Intel lifted the above language directly from the "Features" page of the SAS2xNN dev docs :)

I don't have a source right now, but I definitely remember reading somewhere that the Chenbros can be cross-flashed with the Intel firmware and would subsequently happily identify themselves as Intel cards. Thinking about it, this is extremely plausible because there aren't really too many ways to deviate from the base reference design for an expander. You can change form factors and 8087 vs 8088 ports and PCIe vs molex power supply, but the expander firmware doesn't know about those and doesn't care. Of course, I wouldn't rely on a cross-flash unless I had a Chenbro in hand AND I could afford to brick it ;)
 
Ended up getting the Chenbro.

12377813_721061991368152_501250657092197673_o.jpg

Bench tested ok, I can't tell via the LSI 9207 BIOS or Windows management tool if dual linking is working or not, the initial batch of drives are WAY too slow to show a difference in performance to try and tell that way.
 
What is the Windows management tool you're using?

MegaRAID Storage Manager shows me that my LSI based HW RAID controller is connected to my HP SAS expander via dual SAS links.
 
Where do you see that in MSM? I can see the expander as an entity, and drives hanging off it but nothing indicating a wide link or not between the HBA and expander.

Edit: Looks like they just released an update to MSM yesterday too.
 
Where do you see that in MSM? I can see the expander as an entity, and drives hanging off it but nothing indicating a wide link or not between the HBA and expander.

Edit: Looks like they just released an update to MSM yesterday too.
BnyN2jJ.png


It shows the expander is connected to the raid card on two SAS ports.
 
Last edited:
So, mine just shows as 'enclosure' and there is no connector details listed under Properties for it.

Screen Shot 2016-03-25 at 2.04.14 PM.png
 
I appreciate the heads up, my full name is all over the net tied to this nick name so no worries. :D
 
So, still don't have confirmation if dual pathing is working or not, but I've got OmniOS installed and am getting good perf out of the spinning rust. Sadly, the DC3600s I installed for cache/log duty are only doing 480MB/s, way below what I expected. Time to chase that with Illumos to see where the bottleneck is in the NVMe driver.
 
Back
Top