Looking to add another 16Tb, thoughts?

D

Duncs

n00b
Joined
May 17, 2008
Messages
11
Guess like most people here I've run out of disk space again. When I brought the Infortrend I thought that would be it for a few years, but hey it lasted less than a year before it was full.
It is all used for fairly static data so don't need massive throughput, just reasonable protection with Raid-6 as I don't have a backup.

Here is my current setup:

CM Stacker with 4x 3bay sata storage bays.
Supermicro MB, 2x 3.2Ghz Xeon's + 5Gb Ram
3ware 9500-12 with 12x500Gb WD disks (Formated as 3.18Tb & 1.81Tb)
QLogic iSCSI HBA - Direct connection to Infortrend
Windows 2003 Enterprise
Infortrend iSCSI-Sata Raid-6 subsystem with 16x 1Tb Hitachi disks (Raid-6, 13Tb usable).​
Which every option I go with the Infortrend will stay, untouched.

To get the upgrade going I've already purchased a Supermicro 846TQ Case :D (24 bays) and added a spare Supermicro MB with 2x 3.2Ghz Xeon's, 7Gb Ram + a WD 10K Raptor.

Option 1:
Add a 16 or 24 port hardware array card. (Something like the 3ware 9650, Adaptec 31605 or 52445) But which one and which hard drives? There seems to be so many horror stories around with dropouts, failures, not supported, disable NCQ etc.

Install Windows 2003 advanced, now this is the safe option as I understand windows but only with the right card/disks combination.

Option 2:
The Supermicro MB has 8 SATA ports on board already, add a Supermicro MV8 card with another 8 SATA ports. Add 16x1Tb disks (As they would be direct sata connections and no fussy raid card I would probably use the Samsung or the WD GP ones)

Install Solaris 10, configure the disks with zfs as Raidz2 and share out the pool as an iSCSI Target. Connect this to my existing Qlogic card and format as NTFS.

Now, this is a much cheaper option as I don't need to purchase a Raid Card, but i cannot beleive it is that simple as 2 commands using a free OS. Oh and 1 extra command to add some more disks to the pool if I run out of space.

I've tested it so know it works but must be missing something as nothing is free and it seems too easy. (But then again it is the same software they are pushing with their x4500 server with 48 disks)

Any comments on the zfs route with reliability or the best hardware Raid/Disk combination?

Thanks
Duncan
 
For a system that depended on simplicity/easy to use and performance hardware RAID is great. Pretty much anyone can set it up and be done with it.

For reliability however I'm not sure exactly how hardware raid could beat a software raid... maybe I should be more specific with I'm not sure how a decent 3ware/Areca/LSI controller could be more reliable than software raid in Linux/Solaris.. or anyone with ZFS really, BSD/OSX.

It doesn't seem like there is any way to recover from big failures with hardware raid setups. Maybe someone else can correct me but I pretty much feel once the hardware raid card decides an array is bad you're SOL. Software raid on the other hand you have at least some chance of recovery.

I actually use a 9650 and configure drives as individual disks and then use software raid in linux. Twice now when using software raid I've had an array fail for a non-software related issue and then be able to recover. First time a 5-in-3 module went bad and some of the drives kept powering up and down and killed my array. Second time one drive went bad in a RAID6 but thanks IMO to the 3ware card ended up knocking out drives until the software array got marked as failed. There was a bout 10000 error messages in the 3ware log about all the ports resetting and drives being added/removed that all seemed to stem from a single drive failing and timing out. Anyways got up one morning and array was gone... again no big issue I got it back by marking drives good. Also keep md5sum file and was able to verify no data got corrupted.

Anyways either of these been with hardware arrays I'm not sure what I coulda done if anything. Also its interesting both arrays I've had fail has been due to a bad backplane and a controller flipping out, not because of multiple drive failures.

Also keep in mind ZFS was basically made for what you want to do... its not just random chance it works very well ;). I'd try to scale it up and see how it meets your requirements. If you run into any problems its not a huge deal to switch to a hardware card or something.
 
Duncs said:
The Supermicro MB has 8 SATA ports on board already, add a Supermicro MV8 card with another 8 SATA ports.
Click to expand...
I have three of these, they work quite nicely. Linux support wasn't quite there the last time I tried, but it's been a year and change now.
Duncs said:
Add 16x1Tb disks (As they would be direct sata connections and no fussy raid card I would probably use the Samsung or the WD GP ones)
Click to expand...
I would avoid WD, honestly.
Duncs said:
Install Solaris 10, configure the disks with zfs as Raidz2 and share out the pool as an iSCSI Target. Connect this to my existing Qlogic card and format as NTFS.
Click to expand...
Why iscsi? I would use ZFS natively and share it with Samba or the CIFS server. I'd at least try that method first.
Duncs said:
Now, this is a much cheaper option as I don't need to purchase a Raid Card, but i cannot beleive it is that simple as 2 commands using a free OS. Oh and 1 extra command to add some more disks to the pool if I run out of space.
Click to expand...
Well, it's not quite that simple. Creating pools is dead simple, this much is true, but if you want to add a few disks at a time (one or two, as opposed to eight) you're out of luck so far: you can't yet change the width of a raidz/raidz2 vdev. So if you put together a 16-disk raidz2 vdev (not recommended) you can't turn it into an 18-disk one, you'd have to add a mirrored vdev. What I'd recommend is making a pair of 8-disk vdevs; then if you run out of space you can replace the disks in a vdev one at a time and wait for resilver each time, and the pool will magically grow once you finish.
Duncs said:
Any comments on the zfs route with reliability or the best hardware Raid/Disk combination?
Click to expand...
Well, ZFS has been very reliable in production at work. We've got a few hundred users' mail spools on it, a couple terabytes of research data, and we'll be moving user home directories onto it over the summer. We set limits on the ARC cache via /etc/system (see ZFS Evil Tuning Guide), although I'm not sure we needed to do that---most of these systems have enough memory that they don't need to worry. Also, at home, I've got a couple old disks striped together for my BitTorrent cache, which works just fine. Despite the fact that they're on a Pentium 3 I can upload and download at several megabytes per second and play content off it.

The caveat, and it's a big one, is that we've got backups. Although from what I've seen of the mailing lists sometimes dead pools can be miraculously restored, sometimes pools die and don't come back. This doesn't happen often, from what I've seen, and often bugfixes are in Nevada within a few weeks... but it's something to keep in mind. Have you considered how to back up such a large system?
 
Go with option one.

Looks like I have a follower :D yay!
 
Thanks for the feedback, I've just ordered 8x Hitachi 1Tb disks to get me started on some indepth testing with zfs. But still think I may go with the Hardware raid and Windows 2003 as it will mean 1 less server.

fyi, the reason for looking at the iSCSI option; I need to have the disks appear to be local to the server or else it really slows down my media server when scanning or searching files (My own fault as I wrote the application!!)
 
WOW!... Thats a lot of space. Are you making a porn server to share your MASSIVE collection by any chance??? Or do you just like to never delete your temp files? lol
 
Ripping high def movies so that you don't have to hop out of your comfortable seat every time you want to change your selection....my theory at least.
 
Massacre215 said:
WOW!... Thats a lot of space. Are you making a porn server to share your MASSIVE collection by any chance??? Or do you just like to never delete your temp files? lol
Click to expand...



He's probably signed a contract with the government to monitor our internets. :rolleyes:
 
can you post some pics of your current setup?

16tb in less than a year, is business related or strictly personal?
 
HDTV data streams run about 8GB/hr. 1TB is about 120hrs. 16TB is about 1000hrs.

1000 hrs is 20hrs/week for a year.
 
To be fair, I actually migrated 2 of my old servers on to the 13Tb's and this took up 6 Tb's before I even started.

Fyi, before this current setup I had:-

Chenbro 16 Bay PATA Case with 2 x 3ware 7508 & 16x 300Gb Maxtor (Total 2x 2Tb usable) - 3Ghz P4, 2Gb Ram
Chenbro 8 Bay PATA Case with 1x 3ware 7512 & 8x 320Gb Western Digital (Total 2Tb usable) - 3Ghz P4, 2Gb Ram

Moving to 1 server there was a big drop in electric usage & easier maintenance.

Anyway, just ordered the 3ware 9650-24M8 and another 8x 1Tb Hitachi drives to finish off the 846TQ. I've used many 3ware controllers and they have always been good to me.

Duncan
 
Wow, that Infotrend unit is less expensive than the Promise m610i unit I put together a few months back and has more features.
 
I had everything needed to finish the build delivered yesterday (3ware card, cables, disks etc).

Will post come pictures once cabled....
 
All cabled and booted (The Win 2k3 was already to go on the Raptor)

Switched on and all 16 drives spun up in order, started to create the Raid-6 and I estimate it will take 14 hrs.....

96501.jpg

96502.jpg


Just thinking if I should add another few 1Tb to boost the raid size.
 
wow thats hot. Needs more pics though.

How long is it going to take to initialize those new disks and how full is the array presently?
 
I think I finally there, I wasn't happy that I had 4 empty drive bays so I ordered another 4 Hitachi drvies and they were delivered today.

Just re-creating the Raid-6 but this time with 20 1TB disks - formatted it comes out as 16.37Tb. I estimate it will take about 15hrs this time to init.

I was getting about 700Mb/s with HDtune on the old 16 disk array, but will re-test once this one has finished initialising.

Will post some more pics once it finished.
 
seriously... you should consider donating like... 3 of those hard drives to me... lol

Ok, but in all seriousness, you can never take too many pictures/screenshots.

What exactly do you use all of this space for? Bluray/HDDVD/DVD backups/rips? Customer backups? Or is this the server that supplies the 6gb for every user account on Gmail? lol
 
Here are a couple of screen shots, nearly finished moving the data around. It runs my video on demand system at home....

Music: Sonos System - 1xZP100, 4x ZP80's, 2 wireless controllers + 18" Touchscreen in the hall.

Video: 2x Popcornhour HD streamers running my custom GUI. All very wife friendly.....

screen1.jpg


screen2.jpg


tq8461.jpg
 
Infortrend iSCSI box eh? We sell Infortrend where I work (at a storage distrubutor) - it's nice stuff.
 
You must log in or register to reply here.
Back
Top