• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

SAS expander? Help a noob!

B

brutalizer

[H]ard|Gawd
Joined
Oct 23, 2010
Messages
1,603
I am considering of building a storage system at home. I must use ZFS, because I dont want corrupted data. My data is precious to me.

I want to connect a JBOD disk chassis with 10-20 disks to my Solaris PC. How do I connect them? My Solaris PC will have a LSI 2008 JBOD card. ZFS needs direct access to all disks, with no hw-raid card. So I need to use JBOD. Therefore the disk chassi must be JBOD. Also the LSI 2008 must be JBOD.

In the norco(?) disk JBOD chassi, I put all disks and connect them via SATA cables. What do I do with all SATA cables? Connect them to the LSI 2008 card via SATA? Or, can I connect all SATA cable into something called "SAS expander" and then connect a single cable to my Solaris PC LSI 2008 card?

I dont understand how to connect the chassi to the PC. And I dont understand what a SAS expander does. Can someone please explain a bit to a noob? How can I design this setup?
 
i have norco rpc 4224 and controller cards what i have are HP SaS expander + LSI SAS 9211-8i

I have that LSI card in PCIe-x1 slot and expander in x16 slot... lsi and expander are connected together with 2 sas cables.. and expander connected to norco sas slots.. easy..

read more info from this site http://hardforum.com/showthread.php?t=1484614
 
4 cables, the same as with the expander. Mini-sas cables connect 4 drives at once.

Do you mean you are not putting your motherboard inside the norco case?
 
drescherjm said:
Do you mean you are not putting your motherboard inside the norco case?
Click to expand...
Exactly. The JBOD chassi holds just the disks, and nothing else.

Is this possible to do? Or must I put a motherboard inside the norco chassi?
 
Then you probably want an expander in the norco and 1 external link from the pc to the expander. Make sure you get a SAS 600 card and expander so that you can get 2.4GB/s total bandwidth.
 
Ah, progress! Thanks, I have tried to get this information from many forums, but never gotten a good answer.

So, just to confirm, I need to put an SAS expander in the norco chassi? Then I connect all SATA disks to the expander? And from the expander, I connect a SFF8088 cable to my LSI 2008 card in my PC? Is this correct? I do not need motherboard in the norco?

Another question, the LSI 2008 card I am thinking of, supports 8 sata disks. Suppose I want a 24 disk norco chassi. Do I need an 24 port LSI 2008 version? Or, can the 8 port card support 24 disks with an expander?
 
Another question, the LSI 2008 card I am thinking of, supports 8 sata disks. Suppose I want a 24 disk norco chassi. Do I need an 24 port LSI 2008 version? Or, can the 8 port card support 24 disks with an expander?
Click to expand...

You are just going to use 1 single 4 port sas connector to support all of your drives. So in theory only a 4 port sas is needed. The expander takes the 2.4 GB/s bandwidth of the 1 sas connector and converts this to 20 to 36 ports. In this setup make sure you get a SAS expander that can be powered with a molex connector like this one:

http://www.intel.com/Products/Server/RAID-controllers/re-res2sv240/RES2SV240-Overview.htm

This uses an LSI chipset and converts a single 4 port SAS2 to 5 4 port SAS2 (20 total SAS2/SATA ports).

http://www.newegg.com/Product/Product.aspx?Item=N82E16816117207&Tpk=RES2SV240
 
Ok, this is lot of good info! You are the man I! :)

Another question (sorry for being so noobish)
A) The last link you showed, it is powered by a molex. But, I also see a PCIe connection. Does this mean I must use a motherboard in the norco chassi? Or can the card hang loose in the air, without connection to a motherboard?

"The power connector on the top can be used instead of plugging the card into a slot, but using a slot is generally easier if one is available."
This means, it can hang loose in the air. Is this correct? I dont need a motherboard?


B) And, I also see six connections on the card. Does this mean I use one connection for SFF8088 between the norco disk chassi to the LSI2008 card in my PC? The rest of the 5 connections supports the 20 SATA disks in the norco chassi?


C) Does one cable between the pc and the norco, give 2.4 Gigabyte/sec? Is this true?
 
Last edited:
A) The last link you showed, it is powered by a molex. But, I also see a PCIe connection. Does this mean I must use a motherboard in the norco chassi? Or can the card hang loose in the air, without connection to a motherboard?
Click to expand...

You can use either to power the expander. The molex is there so you do not need to have a motheboard to use the card. You will however have to mount the card so that there is some space on all sides in the norco. I would get some long screws and nuts at a hardware store and mount the card 1/2 inch or so from the bottom of the case in the motherboard area.

Here is the part of the manual that describes this.
5.(Optional) In case power can be supplied remotely from RA 4-pin 12V/5V power connector, the expander card can be located elsewhere within the server chassis instead of using a PCI Express slot, if only the cooling condition can meet the expander card's thermal and atmospheric requirement. The expander card can therefore be fastened by removing its bracket and fasten it to a customized chassis wall (see Figure 3). The expander card already has two holes for fastening the bracket and three additional holds reserved. User needs to prepare screws and standoffs to match the hole size on both the expander card and customized
chassis wall.
Click to expand...

B) And, I also see six connections on the card. Does this mean I use one connection for SFF8088 between the norco disk chassi to the LSI2008 card in my PC? The rest of the 5 connections supports the 20 SATA disks in the norco chassi?
Click to expand...

The Intel expander will support 1 or 2 input connectors. With 1 connector you will gave 2.4 GB/s ( 600 MB/s x 4) total bandwidth for 5 sas connectors (20 drives). That should be enough bandwidth to run all 20 drives at the same time. With 2 connectors you will have 4.8 GB/s bandwidth for 4 sas connectors (16 drives).

SFF8088 is the external connector. For the connection to the expander you need a bracket that connects SFF8088 to SFF8087.

http://www.pc-pitstop.com/sas_cables_adapters/AD8788-1.asp

or if you plan to use 2 links from you lsi hba to the expander for 16 ports with 4.8 GB /s bandwith

http://www.ipcdirect.net/servlet/Detail?no=254

http://download.intel.com/support/motherboards/server/sb/e93121003_res2sv240_hwug.pdf


For the connection from the expander to the norco backplane (where the drives connect) you need SFF-8087 to SFF-8087 cables like this.

http://www.ipcdirect.net/servlet/Detail?no=216

C) Does one cable between the pc and the norco, give 2.4 Gigabyte/sec? Is this true?
Click to expand...

Yes. A sas2 connection is 600 MB/s. So the cable containing 4 of these would be 4 times that being 2.4 GB/s .
 
Last edited:
Holy cow. I think... that answers all my questions?! Thank you guys! I now know everything I need and can start planning my server! :)

@WestSidaz,
As I have understood it, the PE-2SD1-R10 gives you an PCIe slot, without the need of using an entire motherboard. But I dont think it fits my plan. But thank you for the suggestion!

The purpose of a SAS expander is to collect all 20 SATA cables in the norco disk chassis, into one long cable. That long cable is then used to connect the PC with the norco disk chassis. That is the purpose of an SAS expander. As I have understood it.



My master plan
------------------------
Requirements:
A) Use ZFS, because ZFS protects data against Bit Rot and Silent Corruption.

B) I am afraid of lightning. Surge protection does not help against a direct strike from lightning. Lightning gives a couple of million Joule. Surge protection handles 2.000 Joule. My collegaue told me about a direct lightning strike in her suburb. The entire area lost much of the home electronics: TV, computers, etc. They had to buy everything anew.

C) Quiet and energy effective.

D) I dont need access to all my data all the time. I am a home user with a girl friend, not a business.

E) Storage with ~20 disks. This will be a cool and nerdy experiment.

F) I will not use backup



Solution
-------------
A)
-Dont use hardware raid, because that messes up ZFS data integrity.
-Use JBOD => I use LSI2008 controller card flashed to JBOD mode. Because Solaris is compatible with LSI2008.

B)
I will have my permanent disk storage in a norco chassis, a raid with ~20 disks. I will have a separate Solaris PC. It will be a mini/micro chassi. In the PC I will have a 240GB SSD as a boot disk, and a 3TB disk as temporary storage, a cache. On the 3TB disk I will put things from the raid and work. Most of the time, the raid will be powered off, with no cables connected. Thus, lightning can not reach it. When I need new data, I power on the norco chassi and copy things to the 3TB disk, and then immediately power off the raid.

C) D) The raid is powered off 95% of the time, so it is quiet and energy effective. Sometimes I copy data for a few hours, but that happens once per month. The rest of the month, the raid is not connected to power. It is totally isolated. The mini itc PC is quiet and cost effective. It will use Sandybridge, or Ivy Bridge - they dont suck power when idling.

It will also have a Nvidia graphics card, and when I want to play, I reboot into Windows. Windows is installed on an external eSATA disk which I plugin when I need to play. Otherwise, it is not plugged in.

E) I use an external norco disk chassi in JBOD mode.

F) I dont use backup, but the raid is protected against lightning becuase it is turned off.
-I should still have a power surge protection, for the few hours I use the raid.
-ZFS protects my data against bit rot.
-I should use ECC RAM for random bit flips in memory.
-If the raid is turned off, then I can make no mistake to destroy the raid. And ZFS supports snapshots, so I can always rollback in time.
-If the PC crashes or is hit by lightning, I dont care. That is just temporary storage. As soon as I have done my work, I just backup the data to ZFS raid, where it is safe.



Execution
--------------
I need a LSI 2008 controller card flashed in JBOD mode. Solaris likes LSI 2008 chipset.

I need a cheap chassi, with ~20 disks. It must not be too deep. I dont need a motherboard, so it not need to be deep. It will only hold the disks, so basically it can be half the size of norcos. Any suggestions?

I need a SAS expander. This expander, I put it into the disk chassi. It will be powered by an molex, so the expander can hang freely in the air, or better: fasten it somewhere in the disk chassis.

I need some cables. One SFF8087 cable so I can connect the mini itx chassi to the big disk chassi when I need to copy data. This gives me 2.4GB/sec between server and disk chassis - which is ample. After I am done copying data, I just remove the cable. My solution does not need to support hot swap, I can reboot when I need to copy data.

Then I feel safe. :)
 
Yes I know, you should use backup. But I will not do that. It is not easy to backup 20 disks with 2TB.

Hence, no backup. I will trust ZFS. :)
 
I have no experience with ZFS, because after reading a lot about it, I decided I couldn't afford it, precisely because it would have to be on top of the backups I do (of around 20 2TB drives, incidentally). There are quite a few stories of things going horribly wrong.
 
precisely because it would have to be on top of the backups I do
Click to expand...

Needing a backup applies to every OS / filesystem. And certainly applies to users of HW raid. I have seen quite a few HW raid cards corrupt data.
 
Of course. That's what I'm trying to say. ZFS has advantages over RAID, that's why the OP, I, and others, are interested in it. But it doesn't mean you don't need backup, and thus, it would cost me money, instead of saving me some like I was hoping for in the beginning (currently, I have 20 HDD and 20 others to back them up, which to most uninformed people feels like a waste).
 
Aesma said:
I have no experience with ZFS, because after reading a lot about it, I decided I couldn't afford it, precisely because it would have to be on top of the backups I do (of around 20 2TB drives, incidentally).
Click to expand...
I did not understand this. Could you try to explain again? If you want to switch to ZFS, you need somewhere the store data while you reformat, and you dont have that extra space? Is this correctly understood?


There are quite a few stories of things going horribly wrong.
Click to expand...
Oh, can you show some links?
 
ZFS is a good alternative to RAID, but it does not replace the need for a backup, unless the ZFS setup is your backup.

Brutalizer, from your posts it sounds like you are planning to use the ZFS server as a primary storage device, but you don't plan on having a backup? Is this correct?
 
I have 20 2TB drives, and 20 others for backup. So, 40TB usable out of 80TB. With something like RAID-Z3 and 23 drives, I could have the same 40TB capacity, using 17 less drives, that could be used to expand capacity. But I would have to trust ZFS, the hardware, and my abilities, far more than I actually do. With my current setup, it's quite difficult to screw things up, and even if I manage, I'd probably only lose one or two drives worth of data. With a ZFS array (pool) going bad, I could lose everything.

So, backups are still needed, and ZFS would just be nice and an interesting experiment, but cost me more drives and hardware than I already own.

As for examples, I found them on [H] topics about ZFS when I was looking into it, I don't have them anymore. I found that one by a quick google : http://christopher-technicalmusings.blogspot.com/2011/02/zfs-data-gone-data-gone-love-is-gone.html

Note that the guy finally managed to recover some data, it just took the help of a ZFS programmer !
 
Yup. Great example, Aesma. Raid is not backup. ZFS is not backup. Modern Raid and/or ZFS coupled with large drives does make the backup challenge more daunting - but it does not eliminate the need. They also allow for massive datastores where the damage from a failure is also massive.

If you value it, back it up.
 
@Neb,
Actually, the JBOD chassi will be my backup. I will use an 240GB Vertex 3 SSD disk as boot disk, and a single 3TB disk as temporary storage. When I am done working, I will copy everything to the JBOD chassi - which will act as backup. My primary storage is the SSD + 3TB disk.



@Aesma,
I trust ZFS more than I trust hw-raid. There are MANY stories about hw-raid corrupting data. Of course ZFS is not bullet proof. It is quite immature yet, soon 9 years old. It takes many years before most bugs are gone from a filesystem.

If a kernel crashes, you loose a couple of hours of work.

If a filesystem crashes, you loose several years of work. A filesystem is much more important to be bug free. A filesystem is more crucial than a kernel to be bug free. When btrfs v1.0 arrives, it will take at least 10 years before people start to trust it seriously.


Regarding the link you posted. He managed to get back almost all his data, yes. And, he use hw-raid together with ZFS - which is a major no no. ZFS can not guarantee data integrity if you use hw-raid controller card. This is well known to ZFS people. Still he did not care, out of ignorance maybe. He did not know?

And he rebooted the Solaris server numerous time while repairing his raid, he resilvered a broken disk and did not wait until repair to finish. When you break one disk, and resilver your raid - do you reboot several times without waiting the raid to finish repair?

And he used deduplication - which also is problematic and buggy right now (for instance, it requires at least ~1GB RAM for every TB disk, say 32GB RAM for 20TB disk). And then he used ESXi pass through, which you should be careful with:
http://en.wikipedia.org/wiki/ZFS#Data_Integrity
Read the quotation at the very bottom in this paragraph.

One guy lost his entire ZFS raid, because he used cheap disk controller together with VirtualBox. VirtualBox does not honour standards, just read the link above and you will see. You have to tweak VirtualBox to honour standards for ZFS to work. I would not use any virtualization with ZFS, no. And, I use snapshots, which the guy did not do.

Many levels of misuse I would say. If he had been using server hardware from say, Oracle, which is tuned to ZFS, then I strongly suspect he would have had those problems.

But still, ZFS is not 100% bullet proof, but it is more bullet proof than any other solution - in my opinion. There are known cases (mostly old) where people lost their data. Mostly, it has been a bug or misuse from the user.

Besides, you can install FreeBSD/Solaris in VirtualBox and play with ZFS. You create a few empty files, and build a ZFS raid from the empty files. ZFS does not care if you build a raid from discs, files, filesystems or whatever.
 
Last edited:
brutalizer said:
But still, ZFS is not 100% bullet proof, but it is more bullet proof than any other solution - in my opinion.
Click to expand...

That depends on how you define "bullet proof", and the reality is everyone defines it differently, and usually based on how they use their storage solution. If we look at it from the standpoint of which is easier to reverse-engineer to recover data, the edge goes to HW-RAID. ZFS has much more of an abstraction layer to it, and finding an expert to dive into ZFS internals to do a manual recovery recovery is infinitely more difficult than finding someone to analyze the block pattern and rev-engineer a broken set of HW-RAID disks and manually fish the data off. That includes professional data recovery companies, they'll normally tell you you're SOL with a broken ZFS pool because they don't have people to reverse-engineer it.

Regardless, use whatever you think is best for you - because in actuality one isn't superior to another, they're simply different. It's not a Mac vs. PC debate. ZFS does have an edge in also being the filesystem and can do things like file checksumming to protect against silent data corruption, but then loses out in manual recoverability. HW-RAID should eventually become filesystem-aware and be able to perform similar (vendors already working on this to get TRIM for RAID implemented). ZFS and HW-RAID are both striping systems, which is a performance and uptime multiplier and not a longterm infallibility multiplier - striping is actually a detriment to longer term storage since the disks are interdependent. The more fault-tolerant longer-term storage system would be a non-striping scheme like JBOD + an infinite number of parity disks.

Bottom line, no matter which system you choose it will be a compromise. I recommend people spend time with as many different systems as they can (I've been running ZFS/HW-RAID/JBOD+PARITY for years) if they want a more thorough understanding of where the shortcomings are, to be able to better choose what fits their application.
 
Last edited:
drescherjm said:
I would consider skipping the expander altogether and getting a 16 port lsi card + motherboard ports.

http://www.google.com/products/cata...a=X&ei=sANZTpWQJ-XY0QGpu9mhDA&ved=0CGAQ8wIwAA
Click to expand...
You said
"4 cables, the same as with the expander. Mini-sas cables connect 4 drives at once."

So, in the JBOD chassi, I connect four SFF8087 Mini-sas cables to all 16 disks. Then those cables will hang outside the JBOD chassi, so I can connect them all to my Solaris PC with the 16 port LSI card. Is this correctly understood?

A problem is that I need to attach/detach my Solaris PC to the JBOD chassis sometimes. So, I would like a JBOD chassi with disks lying in the basement. And then I pick up my mini-itx chassi and go down to the basement and plug the four Mini-sas cables in and copy. When done, I unplug the cables and go up with my PC and bring the PC to my friend. Is there any easy way to plug/unplug mini-sas cables?


EDIT: I think your suggestion suits me best. It is most elegant, because then I dont need an additional expander. Thank you for your help!
 
Last edited:
Ok, I think I have found the perfect card for my needs. The old version works well with Solaris / Linux according to this site:
http://blog.zorinaq.com/?e=10

Here is the new, upgraded card with external connectors
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118147&Tpk=lsi sas 9201-16e
Thus, I can just plug the cables from the JBOD into this card which sits in my PC.

Here is the version with internal connectors. It is cheaper.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118142&Tpk=lsi 9201-16i


So, with the external version I just plug in the cables into my Solaris PC, power on the JBOD chassis and copy. Then I shut down, and release the cables, then I can remove my PC from the basement.


EDIT: here are more cards suitable for this:
http://www.lsi.com/products/storagecomponents/Pages/HBAs.aspx

Here is the manual
http://www.lsi.com/downloads/Public...pters Common Files/6Gb_SAS_HBA_User_Guide.pdf
Ok, one of the 16 disk cards support PCIe x16 slots (the 9202-16e because it has two LSI chips, each one using x8) and the rest supports PCIe-x8 slot because they have a single LSI chip.
 
Last edited:
There is lies my issue as well. I see ZFS and all these other methods as trying to avoid a true backup. Must be something about WHS freakout mindset of needing 2x the space.
 
brutalizer

I will try to get back to you soon on this. Your questions require some thought on my part and I do not have a lot of free time since I am working 80+ hour weeks at the day job.
 
drescherjm said:
brutalizer

I will try to get back to you soon on this. Your questions require some thought on my part and I do not have a lot of free time since I am working 80+ hour weeks at the day job.
Click to expand...
Ok, but I think it is ok. You have provided me more help than anyone. You can let this go. I am now on track, you have given me a critical mass. I can now learn myself.
 
Ok, I am strongly considering this
- Norco 4224 which I will use as JBOD chassi
- LSI2008 card (the IBM m1015 which is a cheap rebranded LSI2008) with two SFF8087 ports
- SAS expander Intel RES2SV240 with six SFF8087 ports and is fully 6Gbit. HP's card is 3Gbit.
- Solaris 11 Express or OpenIndiana

A) Can anyone confirm that all these work together? Does Solaris 11/OpenIndiana see and use disks connected to the SAS expander?

B) The Norco has 24 disks. I will connect one SFF8087 port from the IBM m1015 card to the SAS expander, which means the SAS expander can only support 20 disks (because it has only 5 ports left). However, the other port from the m1015 can support the rest of the 4 disks. In total the m1015 + SAS expander can support exactly 24 disks using two cables from the PC to the norco chassis. Is this correct?

C) Because the Intel SAS expander card can be driven by a molex, I can let the SAS expander hang in the air in the Norco with no need for motherboard. Correct? (I must see that it gets decent airflow so it wont overheat)

D) The single SFF8087 cable from the m1015 to the SAS expander will give me 2.4GB/sec, evenly distributed among the 20 disks (that is 120MB/sec for each disk). The second cable from the m1015 to the norco backplaine mini sas slot will give me another 2.4GB/sec, evenly divided among the rest of the 4 disks. Is this understanding correct?

E) Can someone confirm this setup works with disks > 2TB?


FYI, you should be careful with SAS expanders and SATA disks when using Solaris, as that is potentially a bad combo. Ex Sun Solaris hacker, founder of Illumos (Open Source Solaris Kernel fork) which OpenIndiana is based on, explains that "SATA disks with SAS expanders are highly toxic" when there is a high load on ZFS disks:
http://gdamore.blogspot.com/2010/08/why-sas-sata-is-not-such-great-idea.html
http://gdamore.blogspot.com/2010/12/update-on-sata-expanders.html
The keyword is HIGH load. My home system will not have a high load, it will only be me. Not many users stressing the disk array doing different things. I hope SAS expander + SATA disks is safe, in my scenario.

PS. I have learned a lot from this forum. I thank all you guys posting so much good info! It is a very good forum. Hopefully I can start to contribute some day, start answering questions, helping people, etc.
 
Last edited:
Ok, I think I am going for it. Just because of this forum. People here inspire me to go for a bigger server than first anticipated. :)


I have a question I hope someone can help me with. I am going for a Chenbro CK23601 SAS expander with an IBM m1015. And a norco chassis 4224 with 24 disks. I am going to use the Norco chassis as JBOD, with no mobo. I will power the SAS expander via a molex.

My question: how many cables do I need, and what cables?
A) 1 SFF8088 between the sas expander and the m1015
B) 6 cables, each having 4 SATA connections to connect all 24 disks to the backplanes. What is the name of these cables? "fan out 4 minisas"? I need to order these six cables, and what should I write in my order? SFF8087? Is there a special brand that is prefered, or are all mini SAS cables of decent quality?

Do I need other cables?

I am thinking of 5 meter for A) and 50cm for B). It should work, yes? Or will I need 70cm for B)?



UPDATE; For 24 disks JBOD, a 350Watt PSU should suffice, yes?



UPDATE 2: I should try to get 120mm fans, right? I am a home user. So I will get three 120 mm fans instead of the stock 80mm fans. What else to think of?
 
Last edited:
brutalizer said:
Ok, I think I am going for it. Just because of this forum. People here inspire me to go for a bigger server than first anticipated. :)


I have a question I hope someone can help me with. I am going for a Chenbro CK23601 SAS expander with an IBM m1015. And a norco chassis 4224 with 24 disks. I am going to use the Norco chassis as JBOD, with no mobo. I will power the SAS expander via a molex.

My question: how many cables do I need, and what cables?
A) 1 SFF8088 between the sas expander and the m1015
B) 6 cables, each having 4 SATA connections to connect all 24 disks to the backplanes. What is the name of these cables? "fan out 4 minisas"? I need to order these six cables, and what should I write in my order? SFF8087? Is there a special brand that is prefered, or are all mini SAS cables of decent quality?

Do I need other cables?

I am thinking of 5 meter for A) and 50cm for B). It should work, yes? Or will I need 70cm for B)?

UPDATE; For 24 disks JBOD, a 350Watt PSU should suffice, yes?

UPDATE 2: I should try to get 120mm fans, right? I am a home user. So I will get three 120 mm fans instead of the stock 80mm fans. What else to think of?
Click to expand...

http://www.monoprice.com/products/p...=10254&cs_id=1025410&p_id=8189&seq=1&format=2

You didnt get the norco with the SAS backplane? if not then search monoprice for the 8087 to SATA connectors.
 
You must log in or register to reply here.
Back
Top