sas hba vs expanders?

[wkearney99]

I've got an ESXi setup that currently has an M1015 (in IT mode, jbod) and an Areca 1260 passed-through to an OpenIndiana ZFS server setup. I'm testing which controller will give me better performance for a 6 drive raidz2. Intial tests seems to show them about the same.

I'm a little concerned about how the 1260 handles drives. It passes them through as individual RAID0 raidsets. It failed out two drives and I'm not sure how well ZFS handles that. I passed the 1260 into the openindiana ZFS guest and moved the drives onto it. I then did a bunch of drive I/O with the intention of stressing the drives. At some point two of the drives failed out. The 1260 sent an e-mail alert but it's not very informative. Nor is the /var/adm/messages log in OI. I'd like to get more informative error messages.

I've moved the drives to the M1015 and am in the process of repeating the drive stress testing. Thus far none have failed.

This leads me the question of how to best connect drives in this setup. I'm probably going to run another VM using Windows and would probably want to pass a controller card into it directly. So adding more one controller is pretty likely. I've got a slot free for another x8 PCIe card so I could go with another M1015 or equivalent. I'm dealing with a chassis with 24 internal SATA bays (4 ports horizontally on each backplane, 6 of them high).

But what about using SAS expanders? If I'm just using the drives for media storage it's not like I'm worried too much about throughput. Just which SAS controllers, expanders and consumer 2tb SATA drives are workable combinations? What should be avoided?

 

[brutalizer]

Some say SAS expanders combined with SATA disks should be avoided with ZFS. It was a problem earlier.

Others say that it is not a problem anymore and that the problems has been fixed.

I dont know more than this.

 

[danswartz]

From what I read, in some circumstances, some sata drives could have issues with some expanders under heavy load. It obviously wasn't a hugely common problem. My understanding as to the best guess cause was that SATA drives behind an expander are talked to using STP (sata tunneling protocol) and that some combination of OS driver, HBA, expander and/or drives would have issues under certain conditions.

 

[wkearney99]

Others say that it is not a problem anymore and that the problems has been fixed.

Yeah, that's the impression I've gotten after wading through quite a few posts. Thus the question.

I've got the slots free and could add two 8 port cards without trouble. I'd want to use a second controller anyway for the pass-through option. But the question is then how to deal with the other 8 bays. I'd either add the 3rd 8-port card or use an expander off an existing one. Expanders are not cheap though; likely more than I'd spend on another controller. But using an expander now would give me more expansion options for the future. Either through something else in the slot or added to the expander (presuming use of one with support for more than just 8 devices).

I'd just want to avoid a setup that would make it difficult to stay on top of drive issues. Past issues with controllers not passing SMART and the like come to mind.

And that's what is driving my current concern, the areca 1260 doesn't seem to pass enough data to ZFS in order for me to truly tell why the controller decided to fail not just one, but two drives. Drives that seem to be working just fine when connected to a M1015 in IT mode. So using 'dumber' HBA controllers for this setup seems like a better plan. I've got other Areca boards doing their job perfectly well in other setups that use the onboard hardware RAID, so it's not like I'm displeased with them overall.

 

[cantalup]

expander could be more expansive than HBA Card.

if you have an empty PCI-express slot, my suggestion: add an extrat HBA card, and does not plan to add more than 16 HDs.
and
you are worry free of expander "issue".

I just bought dell H200 for $75 free shipping.
I learn from someone(stanza... posted the reply and the link to IT or IR flash for H200) that H200 is M1015 alternative where is overpriced recently.

 

[_Gea]

expander could be more expansive than HBA Card.

if you have an empty PCI-express slot, my suggestion: add an extrat HBA card, and does not plan to add more than 16 HDs.
and
you are worry free of expander "issue".

I just bought dell H200 for $75 free shipping.
I learn from someone(stanza... posted the reply and the link to IT or IR flash for H200) that H200 is M1015 alternative where is overpriced recently.

16 disks?

use the foollowing case
http://usa.chenbro.com/corporatesite/products_cat.php?pos=24

insert up to 50 sata disks, each 4 TB to have 200 TB max raw capacity
use 3 x LSI HBA SAS 9201-16

You can even build 200TB highspeed singlecase storage with 50 disks without expander.
You need expanders only to connect several JBOD boxes.

 

[cantalup]

16 disks?

use the foollowing case
http://usa.chenbro.com/corporatesite/products_cat.php?pos=24

insert up to 50 sata disks, each 4 TB to have 200 TB max raw capacity
use 3 x LSI HBA SAS 9201-16

You can even build 200TB highspeed singlecase storage with 50 disks without expander.
You need expanders only to connect several JBOD boxes.

I speaked on M1015 and H200. that has 8 ports where H200 is not overpriced... $70-80 range recently.


building 50 disks ? I would not do that. the HD size is getting bigger and so the total amount of HDs.


16 ports is more expensive then 8 ports