Storage upgrade options

B

Blue Fox

[H]F Junkie
Joined
Jun 9, 2004
Messages
11,813
So, I've run into a bit of an issue in my storage upgrade agenda. My current ~20TB array has been full nearly all year and I feel that now is the time to upgrade to larger drives. I figured that someone else here might have some insight as to what my best options are at this point. Cost effectiveness is key for me.

I guess I'll start off with the equipment I have at my disposal and on hand:
  • Areca 1280ML
  • Areca 1680ix-24
  • 2 x Chenbro CK13601 SAS expanders
  • Enough cases and the necessary hardware to accommodate ~90 drives
My original plan was to use the Areca 1680ix-24 in one chassis with 20 drives (Norco 4020) and then have one of the Chenbro SAS expanders in another chassis with 20 drives (also a Norco 4020) connected externally to the Areca card. This would allow me to recoup some of the cost by selling the 1280ML and the second SAS expander and the result would be far more cost effective. I would then have both chassis populated. One would consist of new 1.5-2TB drives and the other of my old 1TB drives so that I can migrate the data.

The problem with that is that it turns out that WD just killed the TLER utility on all their newer drives, so now there really isn't anything that will work with my 1680ix-24 that isn't in the enterprise drive category. With this in mind, I've kinda looked at my options and none of them is exactly what I wanted. Here they are:

  1. Buy enterprise drives.
    • Good: They're more reliable and I get to keep my 1680ix-24/SAS expander setup.
    • Bad: They cost 2-3x as much desktop drives, so they are possibly the most expensive option.
  2. Replace the RAID card.
    • Good: If I can find the right RAID card, I can buy cheap desktop drives.
    • Bad: Limited options that are supported by the SAS expanders. I like the functionality and management options on my Areca cards too. Would probably need to use both SAS expanders as well.
    • Thoughts: LSI's new 9260 cards might work if I could find a HCL with drives on it (if you can locate it, please let me know). Highpoint might work as well as oddly enough. They support more drives than Areca does even though they both use the came processor (which I thought brought the incompatibilities).
  3. WHS.
    • Good: I can sell all the RAID cards to recoup some of the cost.
    • Bad: I have to buy twice as many drives, and would still need some sort of HBA to support the SAS expanders, so it won't be cheap either. Limited functionality compared to Windows Server 2008 (what I currently use) and no domain support.
  4. Multiple servers.
    • Good: I can use my Areca cards and use cheap desktop drives.
    • Bad: I have to use multiple RAID cards (can't sell one to recoup $1k) and not all the data is in one place since the SAS expanders wouldn't be supported (the 1280ML works with many more drives and I would have to buy a second).
Anyone have any ideas that might be better than the options I've thought of or why one is better than any other? If there are also any large capacity (over 1TB) desktop drives that work with my 1680ix-24, I would like to know. I think the 2TB Hitachis might, but I have yet to have any confirmation and I'm not going to risk buying 20 of them just to find out that they don't. Thanks for getting through my long post! :)
 
First off I want to say stay away from the deathstars....srsly they still suck.
I bought three at frys the other day 2 out of 3 were DOA.

No one said you had to use WD drives.
You could get some Seagate Barracuda LPs. I have 4 of them and I dont have any issues.

My suggestion is a Combination of the above.

Server 2008 R2 w/ WHS Hyper-v VM.
Sell Both RAID Cards
Buy a 8 port SAS HBA non-raid. Im thinking a SM SASLP-MV8 would work, but i dont know if it supports expanders. Bring one of your Chenbros by and we can test it.
Take one 4020 and do a full build.
Install HBA and 1 Chenbro
Populate with 2TB HDDs

Take second Norco
Install 400w PSU
Install second Chenbro
Add current drives and then replace as needed with higher capacity.
 
Id say do your original idea with the 1680ix-24, expander and 2 Norco's but with Seagate desktop drives.
 
Well, WD was the only option left that worked with the 1680ix-24 until WD killed the TLER utility. Funny enough, the second Norco chassis is exactly how you described. I have a 400w Corsair PSU set for it and a mITX Celeron board to make turning the system on/off easy.

I have 20 x 1.5TB WD drives on their way at the moment, but they can be returned easily (I found out about the TLER thing a little late). The main issue with the WHS setup is the fact that I would pretty much have to buy another 20 drives to achieve the ~30TB of storage that I wanted. RAID winds up actually being cheaper.

To be honest, I don't think I'm ever going to find a medium where I'm completely happy, which is why I'm still researching RAID cards. I'm going to give LSI a call tomorrow to find the mysterious drive HCL (maybe ATTO too).

If I do go with the WHS route though, I would probably buy one of the Areca HBAs (1300ix-16) so that I could have 3 Norcos for one system (since 2 would be needed to get me the amount of space I want and the third would be for expandability). I already have a full system in it that I like (Q6600 with ECC RAM, dual gig NICs, and two PCIe x16 slots) that is currently being used as my file server, so at least I don't have to build another.
 
jtg1993 said:
Id say do your original idea with the 1680ix-24, expander and 2 Norco's but with Seagate desktop drives.
Click to expand...
Won't work. Those drives are incompatible with it. If it were that easy, I would have done it ages ago as I already have a couple dozen Seagate drives at my disposal. It's the reason why I still have my 1280ML (they work with it) and my 1680ix-24 and SAS expanders have been sitting unused for 6 months in a box.
 
Blue Fox said:
Won't work. Those drives are incompatible with it. If it were that easy, I would have done it ages ago as I already have a couple dozen Seagate drives at my disposal. It's the reason why I still have my 1280ML (they work with it) and my 1680ix-24 and SAS expanders have been sitting unused for 6 months in a box.
Click to expand...

Since WD drives work and they only started to roll out the TLER thing you could possibly find an older batch of WD drives at a retailer.
 
jtg1993 said:
Since WD drives work and they only started to roll out the TLER thing you could possibly find an older batch of WD drives at a retailer.
Click to expand...
Would unfortunately be at a higher cost. Fry's doesn't have the 1.5/2TB WD drives for anywhere near the cost of what I just paid for them (I bought the 1.5TB drives for $86 a pop shipped). I'm also screwed if I have to RMA one as then I'll have a drive that doesn't support TLER and this is something I expect to do as I normally have a few drives die each year (happens when you have a couple dozen).
 
Is the 1680 really that much better than the 1280 to make you not want to sell the 1680?

If you use the 1280ml then you dont have compatibility issues.
 
One other advantage going with WHS is that it will allow you to purchase only as many drives as you need initially and then buy more as needed (and as prices drop). No expanding array stuff or need for the expensive RAID cards.

Does 100% of what you have need to be duplicated? Are you going to be okay with non-RAID array performance (which should be fine over gigabit)?

For density I'd suggest all 2tb drives, or going to 40tb from 30tb per box later is going to be a big headache.
 
nitrobass24 said:
Is the 1680 really that much better than the 1280 to make you not want to sell the 1680?

If you use the 1280ml then you dont have compatibility issues.
Click to expand...
The 1680ix-24 lets me use expanders. In terms of performance, I don't care which I use, I would just need to have 2 x 1280ML if I want 40 drives (which is naturally more expensive).
[LYL]Homer;1034991673 said:
One other advantage going with WHS is that it will allow you to purchase only as many drives as you need initially and then buy more as needed (and as prices drop). No expanding array stuff or need for the expensive RAID cards.

Does 100% of what you have need to be duplicated? Are you going to be okay with non-RAID array performance (which should be fine over gigabit)?

For density I'd suggest all 2tb drives, or going to 40tb from 30tb per box later is going to be a big headache.
Click to expand...
I want redundancy of some sort for all the data. It actually turns out to be cheaper to have a $1k RAID card than buy double the number of drives (for the amount I want). As for performance, as long as it can saturate at least 1gbit (I have multiple systems with 2gbit connected to a managed switch), then I'll be fine.
 
Well then you are kind of SOL.

If you returned the 1.5s you just bought, and sold your 1280 and 1 expander you could prob buy 10x of the 2TB RE4's
So that would double your capacity.
And you would come out about even.
 
even if the LSI 9620 has the desktop drives on the HCL wont you still have TLER issues?
Plus you can only use up to 32 drives on the 9620-8i
 
Last edited:
nitrobass24 said:
even if the LSI 9620 has the desktop drives on the HCL wont you still have TLER issues?
Click to expand...
They might have other brands on there. I've actually been using Seagate pretty much exclusively until now and would prefer to continue using them.
 
You could take a look at the Adaptec 52445 compatibility .pdf and see if your expanders are on it; the seagate LP drives certainly are on there. You could also look at running multiple 5805's, which don't have on board expanders that the higher end boards do so generally have better compatability. Or you could try trading that 1680ix for another 1280ml (either two servers or a bit of case hacking and one server with two cards in a huge case)

As for LSI, you'll probably have to email their support because I haven't been able to find an HCL linked to on their site.
 
The Adaptec 5445Z and 52445 do seem to have everything I want as they support both the drives and expanders (too bad they don't have out-of-band management). I've only heard bad things about Adaptec when it comes to having issues in the long run however. I guess I'll keep them in mind however.
 
Adaptec = teh suck.

Ive hated everyone ive ever owned.
 
Does that mean the Seagate 2TB LP are addressable and work fine with the Chenbro SAS expander, but just the Areca 1680 card is incompatible with them? Because, as far as I can tell, as long as you jumper them to SATA1 mode, those drives work beautifully with the Highpoint 4320. But I've only had the chance to test a RAID1 2 drive mirror. If you don't jumper them to SATA1, you get a ton of bad sector repaired errors in the log. Whereas on my Area 1231, they work without complaints in SATA2 mode. IMO it doesn't really matter since the drives aren't exceeding the 150MB of SATA1 anyway.
 
P.S. I second that Adaptec sucks. In fact when I was using it, the Adaptec would kernel panic or hard-lock if the Chenbro was connected. So actually you might consider me the captain of the Adaptec hate club. There was a big thread you might dig up right here on [H] with me bitching about their worthless cards. Plus, the Adaptec doesn't support online array roaming, so you can't even safely disconnect/connect raid arrays while the system is online.
 
I guess I could buy a few drives to test temporarily since I already have the Areca and SAS expanders on hand. A couple weeks of heavy 24/7 access (on 4 drives in RAID 6) should do it? If a jumper is all that is needed, then I like that fix. I finally found the LSI HCL and well, it's actually pretty good. I could always get one of the 9280-8e cards and use both SAS expanders.

As for Adaptec, I have yet to find a person happy with them, so, I've been trying to avoid them...and as for Highpoint, well, somehow their 4460 card supports Seagate drives (which is odd as I though the IOP348 brought the incompatibilities, and yet it still has it) and the SAS expanders, but its $1700! I don't know what they were thinking when you can get Arecas for a bit more than half that.
 
You don't have to buy their new top-end card, the 4460 is expensive since it has a lot of ports and its PCI-E 2.0. I'm just using their 4320 (which I got for about $270 when it was on sale at newegg) along with an internal to external mini-SAS PCI bracket, and it's working beautifully to address my external homebrew JBOD (norco 4020 plus chenbro SAS expander and a powerboard to turn on the ATX PSU).

But if the Seagate 2TB LPs work the the 1680 w/ the jumper fix, then that would be the way to go IMO.
 
P.S. like Adaptec, LSI doesn't support array roaming either, they assume you have all the drives you want hooked up from boot-up of the system. So if you ever plan on turning on or off any of the external JBODs while the system is on, you can't do it w/ the LSI. Online array roaming is supported on the Highpoint, however. I think the Areca supports it as well, but I haven't actually tried it.
 
Yeah, if the jumper fixes compatibility issues, then I'll go with the Seagates as I have before. I'm going to go out to Fry's right now to buy 4 drives.
 
Please let us know the results of your tests, if it works it would be an easy way to expand by just adding more JBODs with sas expanders.
 
I'm happy with my adaptec 5805, although I don't need expanders or external arrays and like I mentioned earlier the 5805 has much better compatibility than the 51245-52445 range due to hardware differences.
In fact my only complaint now is the boot time (I use it in a workstation, not a server).

Would you mind posting a link to the LSI HCL?
 
alamone said:
Please let us know the results of your tests, if it works it would be an easy way to expand by just adding more JBODs with sas expanders.
Click to expand...
Will do. It will probably be a couple weeks though.
novadude said:
Would you mind posting a link to the LSI HCL?
Click to expand...
6gbit controllers: http://www.lsi.com/DistributionSystem/AssetDocument/MegaRAID_Interop_List_SAS2_Nov1209.pdf
3gbit controllers: http://www.lsi.com/DistributionSystem/AssetDocument/MegaRAID_InterOp_Nov1209_SAS1.pdf
 
Well, so far I've had very sporadic reliability when it comes to the SAS expander support with the drives. I haven't determined what the problem is, but drives will randomly show up as failed when powering on the system. Unplugging the SAS expander and then plugging it back in usually winds up with another random drive showing up as failed and it takes multiple attempts to get it completely working. Since I have 2 SAS expanders on hand, I'll figure out what the problem is while the array is initializing. Also, finding those jumpers is a pain since they aren't included anymore. I'm glad I kept my old ones since they are so small and not the size of the usual drive or board jumper.
 
Doesn't sound promising; I had similar drive detection problems with the Seagate 1.5TBs (the 7200 rpm ones) on SAS expander, but the Seagate 1TBs worked fine. Have you considered the Samsung Ecogreen 1.5TBs? They work fine w/ my Chenbro, I can create a RAID5 fine, but the Highpoint doesn't like making a RAID6 out of it for some reason - it locks up during initialization. Maybe the Areca would fare better.
 
Correct me if I'm wrong, but the issue with the drives was having time-out errors, right?
 
If you mean the Seagate 1.5TBs, it's just that they wouldn't even get detected even though they were plugged into the SAS expander, or maybe only one drive out of several would get detected.

If you mean the Samsung 1.5TBs, they would just fail initializing midway, but only for raid6. I'm using a 4-drive RAID5 on the sas expander just fine.
 
Well, after all of them were detected (took a few tries), I created a RAID 6 array and one of the drives timed out about 90% of the way into initialization. This is still with the Seagate drives.

Here are the logs:
Code: 
Time 	Device 	Event Type 	Elapse Time 	Errors
2009-12-03 16:45:46 	192.168.001.111 	HTTP Log In 	  	 
2009-12-03 09:39:49 	ARC-1680-VOL#000 	Complete Init 	006:04:19 	 
2009-12-03 09:01:04 	Enc#3 SLOT 011 	Device Failed 	  	 
2009-12-03 09:01:03 	Raid Set # 000 	RaidSet Degraded 	  	 
2009-12-03 09:01:01 	ARC-1680-VOL#000 	Volume Degraded 	  	 
2009-12-03 09:00:56 	Enc#3 SLOT 011 	Time Out Error 	  	 
2009-12-03 09:00:43 	Enc#3 SLOT 011 	Time Out Error 	  	 
2009-12-03 03:38:43 	192.168.001.102 	HTTP Log In 	  	 
2009-12-03 03:37:49 	192.168.001.111 	Telnet Log In 	  	 
2009-12-03 03:36:24 	E3Power 01 	Removed 	  	 
2009-12-03 03:35:30 	ARC-1680-VOL#000 	Start Initialize 	  	 
2009-12-03 03:35:28 	Enc#3 SLOT 008 	Device Inserted 	  	 
2009-12-03 03:35:28 	Enc#3 SLOT 011 	Device Inserted 	  	 
2009-12-03 03:35:28 	Enc#3 SLOT 009 	Device Inserted 	  	 
2009-12-03 03:35:28 	Enc#3 SLOT 010 	Device Inserted 	  	 
2009-12-03 03:35:27 	Enclosure#3 	Added 	  	 
2009-12-03 03:32:54 	000:0000058B8300 	Lost Rebuilding/Migration LBA 	  	 
2009-12-03 03:32:54 	000:0000058B8200 	Lost Rebuilding/Migration LBA 	  	 
2009-12-03 03:32:54 	H/W Monitor 	Raid Powered On 	  	 
2009-12-03 03:32:24 	Enclosure#3 	Removed 	  	 
2009-12-03 03:32:24 	Enc#3 SLOT 010 	Device Removed 	  	 
2009-12-03 03:32:24 	Enc#3 SLOT 009 	Device Removed 	  	 
2009-12-03 03:32:24 	Enc#3 SLOT 011 	Device Removed 	  	 
2009-12-03 03:32:23 	Enc#3 SLOT 008 	Device Removed 	  	 
2009-12-03 03:31:12 	192.168.001.102 	HTTP Log In 	  	 
2009-12-03 03:03:31 	E3Power 01 	Removed 	  	 
2009-12-03 03:03:19 	ARC-1680-VOL#000 	Start Initialize 	  	 
2009-12-03 03:03:17 	ARC-1680-VOL#000 	Create Volume 	  	 
2009-12-03 03:02:41 	Enc#3 SLOT 008 	Device Inserted 	  	 
2009-12-03 03:02:41 	Enc#3 SLOT 011 	Device Inserted 	  	 
2009-12-03 03:02:41 	Enc#3 SLOT 010 	Device Inserted 	  	 
2009-12-03 03:02:41 	Enc#3 SLOT 009 	Device Inserted 	  	 
2009-12-03 03:02:40 	Enclosure#3 	Added 	  	 
2009-12-03 02:58:31 	H/W Monitor 	Raid Powered On
 
Last edited:
It's hard to tell whether it's an issue with the SAS expander or with the Areca. Have you tried direct-connect, bypassing the SAS expander?
 
I guess I'll do that next (though I should mention they'll still be connected via an internal SAS expander on the card itself). I really need to get another Norco for this as I don't have hot-swap capability on the case I'm using for testing...
 
what's the latest status BlueFox? still stuck in "analysis paralysis" mode?
 
Last edited:
Ordered a LSI 9280-8e a couple days ago. It hasn't been delivered yet (FedEx says tomorrow). Hopefully it will get the job done. If it doesn't, I guess I'll just stick with the Areca cards and buy some Hitachi drives like you (and then switch to one of the 1880 series cards when they come out).
 
Last edited:
The good news is you can sell the WD's on ebay or here for more than what the Hitachi's should cost you -- I actually broke even by re-selling my WD's at the same price I bought them for, then grabbing 20 Hitachi's for $141 at Fry's.

I wouldn't count the WD's out yet though- they're good drives, and they sure run cooler than the Hitachi's - OTOH the Hitachi's support lower-RPM APM setting. Let us know how the LSI goes.
 
odditory how long have you been running those Hitachi's?

I had 3 of 4 dead within a week of installation, so that left a bad taste in my mouth.
I ended up using the Seagate LPs instead.

Also BlueFox if you sell the WDs send me a PM i might be interested in a few.
 
Well, the card is here and I've had nothing but problems so far. First off, I hate the LSI storage manager. It is slow as hell (and Java based). I guess I'll run down the list of problems.

  • The drives on port 5 of the SAS expander appear as if they are not attached to the enclosure and are instead directly attached.
  • From the 4 drives just listed above (attached to port 5), two of them show as having the same port number. This is problematic because it results in neither of the two drives being usable when it comes to creating a RAID array with them.
  • If I try to initialize the array, approximately 75% of the drives then immediately fail.

I also get a ton of errors during the process. I already have 3500 lines in the log file. Here is what stands out:

  • Controller ID: 0 Phy is bad on enclosure: Port A:1 PHY 0 (repeat for PHY 1-3 as well)
  • Controller ID: 0 Unexpected sense: PD = Port A:1:9 - Power on, reset, or bus device reset occurred, CDB = 0x1a 0x08 0x48 0x00 0xff 0x00 , Sense = 0x70 0x00 0x06 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x29 0x00 0x00 0x00 0x00 0x00 (happens too many times to count with slight variations...this one happens right before the array dies as well)
  • Controller ID: 0 Unexpected sense: PD = Port A:1:15 - Information unit CRC error detected, CDB = 0x28 0x00 0x00 0x04 0xaa 0x40 0x00 0x00 0x40 0x00 , Sense = 0x70 0x00 0x0b 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x47 0x03 0x00 0x00 0x00 0x00 (also happens a ton of times with slight variations)

I think I'm going to call LSI support tomorrow to figure out what is up...sigh. At least they have 24/7 phone support.
 
You must log in or register to reply here.
Back
Top