Storage/Raid Questions

[tycoonbob]

So let me start by saying this isn't your typical raid questions.

I am new here, but am frequent over on OCN for the past few months, and have started a new storage build.
http://www.overclock.net/t/1272033/nus-server-and-mdc-build-log

Norco RPC-4224
ASUS P8B-E4/L Motherboard
8GB RAM (will be 16GB)
LSI 9621-8i (with BBU)
2x60GB SSDs (Raid 1) - OS
Windows Server 2012 (RC Currently--will be regular release once it's released)

Should have my controller next week, and that will leave my HP SAS Expander as the last thing to get (other than SFF-8087 cables). This box will be using iSCSI with MPIO (4 NICs) to provide storage to a Failover Hyper-V Cluster, with 15-25 VMs. It will also be used as a NAS for data (irreplaceable [photos, home videos, files], 720p/1080p movies to stream on my LAN, 720p TV Shows, Music, and off-site backups for friends/family members). So this is technically a NUS server. Once built, here is my backup plan:
-My current storage solution is 4x2TB HGST Deskstar 7K3000 drives in Raid 5. I will use these 4 drives in a non-raid stup in an eSATA or NAS box, to do nightly incremental backups of irreplaceable data, and weekly offline backups of important Vms (Domain Controllers, Exchange server, etc) using Microsoft System Center 2012 Data Protection Manager.
-Paid subscription to Crashplan, which will allow for unlimited online, off-site backup. Will take forever for first back up, but with a 5mbps upload, I don't care.

Lastly, let me preface that I am a knowledgeable IT Professional working as a Systems Engineer Consultant for a Microsoft Gold Partner (Partner of the year 2012 - Heartland region). I'm a big Microsoft guy, with great working experience of System Center 2007 and new products, and I will be using all the System Center 2012 products (minus Service Manager), as well as Exchange, AD DS, RDS, and whatever else I fancy. This is a home personal build. I am not running any business, or anything like that. If you view the link above, it details my mDC (miniDataCenter) build for my server room, and everything will be racked in a 45U rack, with Battery Backups.

I have four questions:
1.) I want to do Raid 50. Ideally I would like to do 3 7x3TB Raid 5 arrays stripped, and have 3 hot spares in my Norco RPC-4224. However, I won't have the money to buy 24 drives up front, so I want to know if anyone knows and/or has experience with raid expansion on the LSI 9621-8i controller (LSI 2108 chip, same as the Dell PERC H700/H800, and several other LSI cards). Can it expand Raid 50 arrays? I will probably do 2 11x3TB Raid 5s, stripped with 2 hot spares, which means I could start with 6 drives, and hopefully expand 2 drives at a time. Anyone have experience with Raid 50 (or 60) expansions? If the controller can't do this, I will default to a single Raid 6 array with 22 or 23 drives, and have 1 to 2 hot spares. Performance is important, as well as reliability. Does the HP SAS Expander change the Raid Expansion ability (I wouldn't think so)?

2.) I love HGST Deskstar drives...and I want to use the Deskstar 7K3000 3TB drives in my NUS server, but with the recent WDC acquisition, do we know if Samsung is going to keep on the HGST Deskstar series, or are they going to take the tech, and label them Samsung drives? If so, what 3TB would you recommend? I want 3TB, 7200RPM, 64MB Cache...around $200 or less, each.

3.) With doing a Raid 50 or Raid 6 on the LSI 9621-8i controller with a HP SAS Expander, would I see an increase in performance with using two SFF-8087 from the controller to expander (Link Aggregation)? If I did set it up this way, is there any additional configuration, or is it just PnP? If one port on the controller died, would the other still provide connectivity (at one channel speed of course), acting as redundancy?

4.) Does anyone have instructions, pictures, or walk through on how to mount 2.5" drives inside the Norco RPC-4224? No instructions came with it, and I have some additional hardware which seems to be for doing this, but I can't quite figure it out.

Thanks for all your help!

 

[mwroobel]

So let me start by saying this isn't your typical raid questions.

I am new here, but am frequent over on OCN for the past few months, and have started a new storage build.
http://www.overclock.net/t/1272033/nus-server-and-mdc-build-log

Norco RPC-4224
ASUS P8B-E4/L Motherboard
8GB RAM (will be 16GB)
LSI 9621-8i (with BBU)
2x60GB SSDs (Raid 1) - OS
Windows Server 2012 (RC Currently--will be regular release once it's released)

Should have my controller next week, and that will leave my HP SAS Expander as the last thing to get (other than SFF-8087 cables). This box will be using iSCSI with MPIO (4 NICs) to provide storage to a Failover Hyper-V Cluster, with 15-25 VMs. It will also be used as a NAS for data (irreplaceable [photos, home videos, files], 720p/1080p movies to stream on my LAN, 720p TV Shows, Music, and off-site backups for friends/family members). So this is technically a NUS server. Once built, here is my backup plan:
-My current storage solution is 4x2TB HGST Deskstar 7K3000 drives in Raid 5. I will use these 4 drives in a non-raid stup in an eSATA or NAS box, to do nightly incremental backups of irreplaceable data, and weekly offline backups of important Vms (Domain Controllers, Exchange server, etc) using Microsoft System Center 2012 Data Protection Manager.
-Paid subscription to Crashplan, which will allow for unlimited online, off-site backup. Will take forever for first back up, but with a 5mbps upload, I don't care.

Lastly, let me preface that I am a knowledgeable IT Professional working as a Systems Engineer Consultant for a Microsoft Gold Partner (Partner of the year 2012 - Heartland region). I'm a big Microsoft guy, with great working experience of System Center 2007 and new products, and I will be using all the System Center 2012 products (minus Service Manager), as well as Exchange, AD DS, RDS, and whatever else I fancy. This is a home personal build. I am not running any business, or anything like that. If you view the link above, it details my mDC (miniDataCenter) build for my server room, and everything will be racked in a 45U rack, with Battery Backups.

I have four questions:

1.) I want to do Raid 50. Ideally I would like to do 3 7x3TB Raid 5 arrays stripped, and have 3 hot spares in my Norco RPC-4224. However, I won't have the money to buy 24 drives up front, so I want to know if anyone knows and/or has experience with raid expansion on the LSI 9621-8i controller (LSI 2108 chip, same as the Dell PERC H700/H800, and several other LSI cards). Can it expand Raid 50 arrays? I will probably do 2 11x3TB Raid 5s, stripped with 2 hot spares, which means I could start with 6 drives, and hopefully expand 2 drives at a time. Anyone have experience with Raid 50 (or 60) expansions? If the controller can't do this, I will default to a single Raid 6 array with 22 or 23 drives, and have 1 to 2 hot spares. Performance is important, as well as reliability. Does the HP SAS Expander change the Raid Expansion ability (I wouldn't think so)?

Even if you want to do striped arrays, I would suggest RAID6 (with or without SPARE) instead of RAID5. The controller is certainly capable of it and this way at least 2 drive in an array can fail and still maintain uptime)

2.) I love HGST Deskstar drives...and I want to use the Deskstar 7K3000 3TB drives in my NUS server, but with the recent WDC acquisition, do we know if Samsung is going to keep on the HGST Deskstar series, or are they going to take the tech, and label them Samsung drives? If so, what 3TB would you recommend? I want 3TB, 7200RPM, 64MB Cache...around $200 or less, each.

Neither HGST nor Samsung exist anymore. Seagate bought Samsung and WDC bought HGST. The deskstars are done as a product line, closest 3TB drive from WDC is the Red.

3.) With doing a Raid 50 or Raid 6 on the LSI 9621-8i controller with a HP SAS Expander, would I see an increase in performance with using two SFF-8087 from the controller to expander (Link Aggregation)? If I did set it up this way, is there any additional configuration, or is it just PnP? If one port on the controller died, would the other still provide connectivity (at one channel speed of course), acting as redundancy?

Do you mean the 9260i? If you are using 7 drive spinning arrays you will see no appreciable benefit by dual linking for speed reasons. If one minisas failed, depending on what failed (controller or cable) it may or may not stay up with a single channel. It doesn't offer dual path unless you are running SAS with dual ports in any case.

 

[tycoonbob]

Please excuse my lysdexia, but I mean't 9261-8i...as in the LSI MegaRaid SAS 9261-8i
http://www.lsi.com/channel/products/storagecomponents/Pages/MegaRAIDSAS9261-8i.aspx

Which is identical to the 9260, except that the SFF-8087 ports on on the end, instead of the top...which I need since I am actually using an aftermarket cooler on my Xeon.

I am leaning more toward the Raid 6, but the performance gain of the Raid 50 is nice. I know Raid 6 is more reliable, but this is also why with a Raid 50 I would have two global hot spares. Can you confirm or deny that the 9261-8i controller can do online expansion with a Raid 50, adding two drives at a time obviously?

2.) Are you sure? I was under the impression that WD buying Hitachi would still allow HGST to operating as a wholly owned subsidary, still producting HGST hardware.
http://www.wdc.com/en/company/hgst/

I am also not interested in the WD Red line. I want true 7200RPM, not "Intelli-power". I know the WD Reds are designed for NAS, but this isn't just a NAS and I would feel the performance hit when running iSCSI. I want high IOPS.

3.) Even with a 22 drive Raid 6 (two hot spares), your telling me that I would see no performance increase with dual linking?

 

[mwroobel]

1. If you want spanned arrays, with that card you can do R60 (which will accomplish the same you were going to do R50, just 2 more parity drives)
2. As far as we have been told by our rep (we buy a lot of drives), all new drives will be WDC branded. Existing HGST will sell through and be eol'd as they decide.
3. I would never do a 22 drive R6, I would do 2 11 drive striped R6's if you need the throughput. Since you are looking to use 3TB drives, don't go over 14 drives (including spares) per array (rebuild times beyond that offer too much time for additional failures). For the system you are planning on doing, especially if you don't have complete backups of everything the extra 2-4 drives for parity and spare is well worth the extra cost.
If you want a reliable 7200rpm 3TB drive, go with the 7200 3TB Ultrastar from Hitachi or the 3TB Constellation ES.2 from Seagate.

 

[tycoonbob]

Thanks again for your input, however you still haven't answered my question. Can spanned arrays be expanded with this controller.

i.e., if I build 2 Raid 5 arrays, each with 3 disks, and stripe them making a Raid 50...Can I add two drives, assign one drive to each Raid 5 array, and expand my raid 50 volume. A lot of controllers can't do this, and if the 9621-8i can't, then I will be going with a Raid 6.

I am not interested in a Raid60, since that would require 8 drives to start. If I can find 8 3TB drives for $200 each, I'm looking at $1,600 in drives to start my array, which isn't an option. even the Raid 50 would require 6 drives, and be $1,200...but I may be able to swing that.

Your statement on doing 2 11 drive Raid6 I complete agree with, if I had 22 drives to start with...which I don't. If I can't expand spanned arrays with this controller, I will start a Raid 6 and decide how many drives I want to put in it, and build a second array from there (maybe a Raid 10 for irreplaceable data, or maybe a 3 drive Raid 5 (plus hot spare) for my VMs for increased write performance). I am looking for someone who has expanded a spanned array with a controller with the LSI 2108 chipset, hopefully an LSI branded controller.

Thanks for the suggestion the Ultrastar/Constellation, but again...looking for around $200, not $350. I really want to do 3TB Deskstars, but will be tough if they stop producing them, since I am not buying 24 drives at once.

Lastly, I will have full off-site backups of all my data.

 

[kac77]

Thanks again for your input, however you still haven't answered my question. Can spanned arrays be expanded with this controller.

Yes it does. It's in the spec. Not so sure what you mean by spanned arrays when you are talking about raid controllers. Volumes can span arrays, but volumes aren't created by controllers. Either way what you are trying to do will work.

Key RAID Data Protection Features
RAID levels 0, 1, 5 and 6
RAID spans 10, 50 and 60
Online Capacity Expansion (OCE)
Online RAID Level Migration (RLM)
Auto resume after loss of system power during arrays
array rebuild or reconstruction (RLM)
Single controller Multipathing
I/O Load Balancing

I am not interested in a Raid60, since that would require 8 drives to start. If I can find 8 3TB drives for $200 each, I'm looking at $1,600 in drives to start my array, which isn't an option. even the Raid 50 would require 6 drives, and be $1,200...but I may be able to swing that.

You really should do RAID 6/0 when dealing with drives this large. If they are desktop drives you really really want the Raid 6/0. If you don't have the money... save. Or if you have a good backup plan then go Raid 5/0. But if not Raid 6/0 is the only sensible option. If all you have is off site backup.. again you want RAID 6/0.

 

[tycoonbob]

kac77,

I am aware that the description says Online Capacity Expansion, but that doesn't mean expansion of all raid types.

Spanned Raid is not the term I meant, it was actually Nested Raid...as in Raid 10, 50, 60, etc. I'm looking for someone who has experience with expansion of a Nested Raid.

Regardless, I sent a support query to LSI to confirm if the 9261 can do this with a nested raid.

To further clarify, I have actually decided to use 4 Hitachi Ultrastar 600GB 15K600 in Raid 5 for my VMs, and use 20 3TB drives in either a Raid 50 or 60 (if this controller can expanded nested raids), and if not I will end up with 2 Raid 6s, and use Server 2012 Storage Spaces to manage them.

 

[mwroobel]

Actually no, RAID 10, 50 & 60 are spanned arrays. They take identical RAID 1, 5 or 6 arrays and stripe the volumeset across the arrays. Nested arrays would be RAID 66 for example, a RAID 6 of RAID6 arrays. So if you have 8 - 8 drive arrays, you would get the available space of 6 - 6 drive arrays. Each RAID would have 2 parity disks, and you would have 2 parity arrays in the larger scheme of things.
As to your question if you can expand an existing R60 with your card, the answer is yes (I have done it with multiple lsi cards in the past). The procedure is to add a drive to each existing RAIDset in the spanned R60, and then expand the volume (or volumes) that exists on that RAIDset. It is therefore a 3 step process. Expand the first R6, expand the striped R6, expand the volume. I know you answered my question that you have backups, are they onsite or offsite? If they are offsite, trust me when I tell you that you should go R6/60 over R5/50. Forget the R5+spare and just do R6, or even better R6+spare. You will have a much happier life because the first time a second drive drops out during a rebuild of your R5, you lost everything until you send out for your backups and lost everything that happened since the last backup. Especially as dynamic and changing as VMs can be, that can be a huge hole in your uptime challenges.

 

[tycoonbob]

http://en.wikipedia.org/wiki/Nested_RAID_levels

They are actually Nested Raids, but regardless thank you for the explanation about the process, and that you have done this with LSI cards before. My only questions is that from your description, you add a drive to the first RaidSet (Raid 6 #1 in a Raid 60, for example) making it have 5 drives, while RaidSet #2 still has 4...how are these RaidSets still in a Raid 0 when one has more drives than the other? I can see that the controller will see the first RaidSet as a logical volume, and the second RaidSet as a logical volume, and just stripe those...but wouldn't it only stripe the highest common space on each? After you add a new drive to each RaidSet (for example make Raid 6 #1 go from 4TB to 6TB) you will still have a Raid 60 of 8TB, until you expand the Raid 60 to 12TB? I'm more of a hands on learner, and I think I won't fully understand until I just do it...but know that I can do it allows me to plan it out better, and know that I have a reason to save for 8 drives.

My full backups are off-site, but that is the data that is replaceable. The irreplaceable data will be on-site and off-site. I would never do a Raid 5 that is over ~8TB, and was planning on 1-2 Raid 6s...unless I went with a Raid 50. With a real raid controller, it should be doing background scrubs to find and fix any UREs that would cause a rebuild to fail. I know this is more and more risky when talking about larger drives, and larger arrays (considering a consumer grade hard drive is typically rated at a URE or 1 in 10^14 blocks, or every 10TB or so), but I will more than likely save up for 8 drives and do a Raid 60, if I am going to use 15K SAS Ultrastar drives for my VMs. I have no idea why that thought never crossed my mind before, but the IOPS would be great with that. I am, however, going to do a 4 drive Raid 5 with the Ultrastars (4 x 600GB drives), and I will have a cold spare ready, eventually. I will be doing offline backups of all my VMs on a weekly basis, with DPM.

Thanks for the clarification that I was asking for, and I will also be sure to post what LSI tells me once they answer my question.

 

[mwroobel]

I haven't done it with any of the SAS2 cards from LSI, I did it with one of the older cards (and not every feature carrys forward) so please get an authoritative answer from LSI before you make a purchase. As to scrubs, if you are truly worried about bit rot then you can set background checks and scrubs, just set them to do it during downtime because it can put a performance hit on your array while it checks. Again, with the 4 drive R5+spare of the US600's, do a RAID6 instead, since you are losing 50% of the available space in any case, set it up so ANY 2 drives can fail and maintain uptime, where in the R5 only the spare and 1 OTHER drive can fail and maintain uptime. If 1 array drive fails and a second before the rebuild completes you are screwed.

 

[tycoonbob]

Oh my controller and BBU are already on their way. The OCE of a Raid 50/60 is something I have had a question about for the past few months, and I haven't been able to get a definite answer, and I never thought to contact LSI before yesterday.

The 15K600 is actually a 4 drive Raid 5, not a 3+1. I will be doing offline backups of my VMs on-site, so even if one dies and it can't rebuild quick enough, I can just rebuild the array. Heck, I just rebuilt my domain over the weekend to move all my VMs to Server 2012 RC. While it may be a home production environment, it's also my personal test environment for work. I have no problem doing the 4 drive Raid 5 for those, but after thinking more on it last night, I am convinced that I will do a Raid 6 or 60 for the remaining 20 drives. No sense not to, and I'm not trying to max out potential storage capacity anyway.

Raid 60 with 18 3TB drives, and 2 hot spares will still be 42TB, plus 6TB in hot spares, plus 1.8TB on the 15K drives plus 600GB cold spare. Puts me right at 50TB, which is a lot considering I only have 6TB or so of data, currently. Should last me a few years, once it's full of drives.

Anyone know of a photo or a walkthrough on mounting 2.5" drives in a Norco RPC 4220 or 4224 chassis?