Multiple SAS card storage shelf

[JohnH]

We have a Coraid ZFS system and are very happy with it's video streaming performance. Unfortunately Coraid's future is gloomy to say the least. http://www.theregister.co.uk/2015/01/15/coraid_struggling_say_sources/

I'd like to replicate the Coraid success using Nexenta and SAS in place of Oracle and AoE. I believe part of the reason Coraid performed so well is the 5 SAS controllers in the 36 drive shelf. The Coraid shelf is a SuperMicro storage server with 10Gig ethernet in place of SAS for connection to the NAS server. I'd like to use the same storage server architecture in this new system. Unfortunately Nexenta only has expander based JBOD shelves certified.

Has anybody used a storage server based storage shelf with multiple SAS cards as a JBOD? Is this even possible with SAS?

 

[_Gea]

A server with multiple SAS controller is not a problem.

As an alternative you can use
- Multiple JBOD boxes connected via external SAS connectors + expander to your server

This is a solution with a single server + multiple dumb jbod boxes where you only need
the expander to connect the (preferred) SAS disks. Distance between server and SAS boxes
can be up to several meter

Another approach is using storage heads.
This can be a ZFS server with several iSCSI, IB or FC cards to connect the head(s) with
as many storage servers as needed. Each storage server is a complete ZFS server with
disks building a datapool. You can either connect them directly or over switches.
You share the combined pool from your storage head to your clients.
From your storage boxes you share the local pool in whole or in parts as a blockdevice target
ex via 10G iSCSI over direct links, a dedicated SAN network or over your regular LAN

Distance between storage head and storage boxes can be up to several kilometer.

Advantage
Each storage box can be a vdev for your storage head what means that you can scale capacity and
performance with quality and number of storage boxes. If you use a spare storage head, you can switch
services within minutes without the need of a complex HA solution. Think about this solution like
replacing a local ZFS disk with a fast ZFS storage server over your network.


Active/active HA can be an addon.

Initiator support for storage heads is something, I added a to my newest napp-it (Menu Comstar - Initiator)

 

[JohnH]

A server with multiple SAS controller is not a problem.

Thanks for the response. Sorry if I wasn't clear but I'm asking if one can create a JBOD, not a server, using multiple SAS cards avoiding expanders completely. This is how the Coraid shelf is architected. It uses a server populated with 5 SAS controllers each connected to 8 drives max. This is a storage shelf not a server but built with a server essentially with no expanders.

I want to build a 36 drive JBOD shelf with 5 internal SAS cards with two external SAS ports. Is this possible?

Thanks]
John

 

[_Gea]

You can build a server (with a mainboard with enough pci-e slots)
- using up to seven normal sas controllers each with 8-16 Sata disks (no expander)

but you cannot connect such a config via (external) SAS to another server
beside using SAS switches, see http://www.lsi.com/products/pages/lsi-sas-6160-switch.aspx

Another option:
Use a regular ZFS server as a blockdevice (similar to AoE) and share storage via IB, FC or iSCSI
(a whole ZFS server treated like a single local disk)

see storagehead config at
http://napp-it.org/configurations_en.html

 

[JohnH]

...you cannot connect such a config via (external) SAS to another server beside using SAS switches

I was afraid of that. I suppose a SAS switch would essentially be the same, bandwidth wise, as an expander but in reverse.

I was hoping to use the same architecture as Coraid but with SAS only because it's a proven performer for us.

 

[_Gea]

I was afraid of that. I suppose a SAS switch would essentially be the same, bandwidth wise, as an expander but in reverse.

I was hoping to use the same architecture as Coraid but with SAS only because it's a proven performer for us.

AoE (Ata over Ethernet) is an alternative to FC or iSCSI as they are the alternatives to AoE.
Ata, Sata, SAS, SCSI etc are all local options, not over a network.

 

[cookiesowns]

You can build a server (with a mainboard with enough pci-e slots)
- using up to seven normal sas controllers each with 8-16 Sata disks (no expander)

but you cannot connect such a config via (external) SAS to another server
beside using SAS switches, see http://www.lsi.com/products/pages/lsi-sas-6160-switch.aspx

Another option:
Use a regular ZFS server as a blockdevice (similar to AoE) and share storage via IB, FC or iSCSI
(a whole ZFS server treated like a single local disk)

see storagehead config at
http://napp-it.org/configurations_en.html

Why not? Couldn't you just use external SAS connections ? Does this user just want one storage head controlling a few storage shelves?

Maybe I'm misunderstanding?

You would have one storage head doing the ZFS. You have a bunch of HBA's with external SAS connections directly connecting to drives JBOD. It's essentially the same thing as getting one large chassis, vs one small chassis with a few JBOD's

 

[_Gea]

This is a more basic question whether you want to create a ZFS server with direct attached storage (directly connected Sata/SAS disks) or if you need a network based SAN technology like AoE, FC, IB or iSCSI.

If you have one rack with a storage head and some external SAS connectors, you can connect Jbod boxes with a few SAS disks directly but you are very quickly at a point, where you need expanders what the OP does not want.

You are also limited in cable length and number of shelves or in failover options, so I suppose there is a reason that the OP had used AoE and not a basic SAS solution.