18TB: need a solution

[jonathonball]

So I bought 18TB (6 x 3TB) of storage for the server I'm building in my new house.

The hard drives are SATA 6gb/s... after doing a little research I realized this density would have been better served with SAS drives, but the trigger has already been pulled so I'm posting here to see what you guys think will be the best way to configure this.

Primary OS is going to be a bare-metal hyper-visor (probably proxmox) on a small SSD.
I'm hoping for full redundancy (~9TB of usable storage space). I'll be running 3-4 Ubuntu server VM's. I have a few hundred bucks set aside for controller cards.

I also read somewhere that you are pretty much mathematically guaranteed to have an unrecoverable read error after 10^14 sectors using SATA, so that's kind of scaring me.

Any thoughts?

Cliffs:
bought 6 x 3TB SATA (6gb/s) drives (http://www.newegg.com/Product/Product.aspx?Item=N82E16822148844)
need recommendations on which RAID configuration to use
need recommendations on which controller cards to buy

System:
i7 950
ASUS Sabertooth (X58)
12GB DDR3
Crucial M4 120GB SSD
Corsair HX650 PSU

Thanks guys.

 

[m1abram]

Since you are already planning to do a VM, why not use OI+napp-it to build a ZFS array. Then do a strip of mirrors with the disks.

Though to do this it is best that you can pass through the HBAs to the OI vm. For this you may want to pick up an M1015 HBA and cross flash it to IT mode.

 

[jonathonball]

Since you are already planning to do a VM, why not use OI+napp-it to build a ZFS array. Then do a strip of mirrors with the disks.

Though to do this it is best that you can pass through the HBAs to the OI vm. For this you may want to pick up an M1015 HBA and cross flash it to IT mode.

That card runs about $300, which is in my budget, but it only has SAS connectors. I've honestly never done a raid larger than four disks. Will the disks I bought work with that card?

 

[jonathonball]

Also does anyone think it will be worth the time to run badblocks or something similar on these drives before I get started?

 

[drescherjm]

That card runs about $300, which is in my budget,

Usually had for ~$75 on eBay used.

http://www.ebay.com/itm/IBM-ServeRa...sk_Controllers_RAID_Cards&hash=item1c2d5be818
http://www.ebay.com/itm/IBM-ServeRa...sk_Controllers_RAID_Cards&hash=item1c2d9dccba
http://www.ebay.com/itm/IBM-ServeRa...sk_Controllers_RAID_Cards&hash=item2ec3aca0ac

but it only has SAS connectors.

You can get SFF-8087 forward breakout cables (to SATA/SAS) for less than $15 each shipped.

I've honestly never done a raid larger than four disks. Will the disks I bought work with that card?

This is non windows software raid so for the most part regular desktop can be used.

Also does anyone think it will be worth the time to run badblocks or something similar on these drives before I get started?

I do that to every single drive I get new or back from an RMA. I have caught quite a few DOA drives from both.

 

[jonathonball]

You can get SFF-8087 forward breakout cables (to SATA/SAS) for less than $15 each shipped.

Awesome, that's a huge relief. Thanks for the info.

 

[parityboy]

@OP

badblocks is always a good idea, along with regular RAID scrubs. I scrub my RAID 6 array once per week.

 

[sethk]

My recommendation is to get a decent LSI based controller and do hardware RAID 10 for ~9TB usable (or RAID 5 for 15TB usable). The former is a little faster and possibly more reliable, at the expense of space. I'm not sure if you really need that much space for your VMs, or if you just bought it so that you have enough spindles / speed. If you are planning to build a high transaction load database or email server, this is not a great storage setup, BTW.

Some will recommend going the more software / DIY approach, and there are no doubt some advantages there. If you want a set-it-and-forget-it approach, and panic free drive swaps when a drive goes bad, I recommend the hardware RAID approach. Also, I would buy a spare drive and keep it on the shelf. It all depends on what your priorities are.

 

[jonathonball]

My recommendation is to get a decent LSI based controller and do hardware RAID 10 for ~9TB usable (or RAID 5 for 15TB usable). The former is a little faster and possibly more reliable, at the expense of space. I'm not sure if you really need that much space for your VMs, or if you just bought it so that you have enough spindles / speed. If you are planning to build a high transaction load database or email server, this is not a great storage setup, BTW.

Some will recommend going the more software / DIY approach, and there are no doubt some advantages there. If you want a set-it-and-forget-it approach, and panic free drive swaps when a drive goes bad, I recommend the hardware RAID approach. Also, I would buy a spare drive and keep it on the shelf. It all depends on what your priorities are.

This for my home server, it's going to function as a NAS, record home surveillance over IP, run an apache server for code testing purposes, bit-torrenting, and anything else that comes up in the next few years. I'm planning on it basically being a playground to geek out on.

After a lot of great feedback from co-workers, friends, and [H].. I think I'm going the software / DIY approach. I'm really intrigued by ZFS. Everyone in this sub-forum has been very helpful without the normal snarky bullshit than can tend to accompany my half-retarded questions. Thank you everyone.

 

[tafkamk]

I just setup something very similar on ZFS 6x3TB in a raid-z2 setup and I couldn't be happier. Performance is outstanding Bonnie++ reports 630+MB/s Read and 320+MB/s Write and my machine is just pieced together from 2-3yr old hardware lying around (minus the drives).

i5 750
P55 FTW (using onboard sata for 3tb drives, pci-e sata card for boot drive)
16GB (2x8gb - really surprised this worked on the old 1156 platform)
6x Seagate 3TB

Im by no means a ZFS expert but if it helps you out after researching the various options raid-z, mirror and raidz-2 primarily I decided on raidz-2 as it seemed to be the best tradeoff between speed and data risk. All 3 options would allow up to 2 drives to fail but raid-z2 allows it to be any 2 drives where as raid-z or a mirror can experience data loss if the wrong 2 drives fail.

 

[m1abram]

Raidz can only have a single drive failure regardless of which drive the second one to fail is.

Only if you stripe Raidz would it allow more than one drive failure.

 

[tafkamk]

the poster above is correct to clarify i was talking about 2xraid-z(3 drives each)

 

[TheDingy]

I did 24TB using RAIDZ-2 on a basic controller with 12x2TB drives and it worked great. I would put some more money into ram, 32GB and run nas4free.org or something like that, it would work like a champ.

Cheers!

 

[jonathonball]

What OS are you guys using for ZFS? I've read some guides for shoehorning it into proxmox but from what I'm reading ZFS on Linux is sort of half-baked. I'm installing Solaris on my laptop now just to play with it and I lightly hate it just based on the installer.

 

[tafkamk]

Using the latest version of OpenIndiana myself

 

[m1abram]

Openindiania myself as well. It seems to have the best support with Napp-it which is a very nice interface for ZFS.

 

[tafkamk]

Yep Napp-It is an extremely useful (albeit ugly) addon, especially if your less then comfortable in the command line

 

[jonathonball]

Installing openindiana now on my laptop for a test drive.. and (again) just based on the installer, what a difference! it's like night and day. It does my heart good to see gnome2 in action as well.

 

[stevebaynet]

Yup, go with OI+Napp-it, if your running it as a VM just for storage you'll hardly ever touch it once its setup anyways.

 

[TheDingy]

I have never tried Napp-it, how well does it work? I used to use FreeNAS but have been using Nas4Free lately on stuff that I don't have the budget for solaris.