Most affordable 70TB solution?

G

Guldan

Weaksauce
Joined
Jan 16, 2013
Messages
113
Hey Guys,

So my friends business stores massive images and he requires at least 70TB usable storage after redundancy.

First thing that comes to mind is direct attach / chain some MD1000's together and manage it from a cheap dell server like an old 2950. I did this at work for our file server and for myself and it works pretty good. But it isn't cheap especially if new.

He doesn't need high speed, I/O won't be very high and only a couple people will access the server at any given time and it's for big pictures not DB or anything. Backups are taken care of as well.

Any single chassis solutions worth mentioning? Budget is $15k and drives will eat 1/3 of that.
 
AcmeMicro will sell you a nice config for $15k. 2 e5-2620, 36 3TB enterprise Hitachis, 64GB of memory. That's 108TB raw, which means you could go with 4*9-disk raid6 or raidz2 arrays and get a total of 84TB.

What's the IT situation like at his company? Are they relying on you for support? Who's on call when it breaks?
 
don't even read my post, unhappy_mage has it right.


hmmmm will you be using ZFS? do you have power constraints? i.e would like to keep the power bill at x?

5k in drives? 3TB consumer grade drives can be had for 130 a piece, 5k/130 = about 38

38 * 3 = about 116TB

let's say 36 drives and 2 hot spares - not an optimal solution but one that comes to mind

if doing ZFS all software I'd do the following

3 norco 4224 24x3.5 bay chassis about 400 a piece = 1200 total
two 750W corsair PSUs = 2x 100 ish = 200
3 Intel or HP sas expanders = about 250/each or 750 total
3 backplanes if using the hp expander = 65x3 = about 200

a main rig to run it all

3x ibm m1015s - 150 a piece these days 1/per norco chassis

here you could decide to go with some fault tolerance and virtualisation...that's over my pay grade so i'd opt for a beefy dual xeon setup,

maybe using l5520's - 100
96GB ecc ram - 12*8GB - 30x12 = 360
Supermicro board - 150 - ebay
chassis to hold it all - 100 or so
adequate cooling - 50 - 100 or so fans and what not
throw in an intel x799 10-gigabit ethernet card for 300 or so (cheaper on ebay)

freebsd running native and ZFS

ZFS will take care of the raid

each norco holds 12 drives

comes out to about 4k for all of it, but it's no where near all in one end all be all enterprise grade solution that people will recommend. you could use the extra cash to build in redundancy with virtualisation and fail over protection, build out more jbod nodes for the drives, increase redundancy etc

depending on desired fault tolerance you could do nested raidz akin to raid 50 for something like 6 pools of 6 drives with 5 data drives in each pool and one drive as a parity that would net you 5x6 or 30 drives worth of data x 3TB a piece at 90TB, increasing redundancy to a nested group of raid6 or raidz-2 would net you 4 drives out of 6 with data and 2 with parity for a raw space amount of 4*6 or 24, 24*3 being 72TB which will come out to about less than 70 as each 3TB drive is really only 2.7TB...

Just some thoughts, I'm sure it's something similar to what you guys might have been mulling over

FWIW unless you want the risk I would have a pro company build this out for you and buy a service contract. Peace of mind is worth it.
 
Last edited by a moderator:
unhappy_mage said:
AcmeMicro will sell you a nice config for $15k. 2 e5-2620, 36 3TB enterprise Hitachis, 64GB of memory. That's 108TB raw, which means you could go with 4*9-disk raid6 or raidz2 arrays and get a total of 84TB.

What's the IT situation like at his company? Are they relying on you for support? Who's on call when it breaks?
Click to expand...

He doesn't have an IT team as it's a small company. Which is why i'm leaning towards just getting brand new Dell stuff but that would put him way over $15k. Also I prefer ZFS but am leaning towards Windows as it is what he knows and easier to manage by non-IT people.

Gigatexal:

We haven't been mulling over much as he isn't exactly tech saavy, I'm just trying to at least point him in the right direction but might end up building this for him if it's not too complex.

One thing you did bring up which i've been considering is 10Gig connections, do you think this is required? I'm just worried it will cost a lot of $$ and not be utilized fully.

My work file server = (perc6e->md1000/md1000/md1000) at work and the server powering it has 12GB of ram (It's a cheap dell w/ a perc6e raid card). Wouldn't I only need large amounts of ram for Software raid and/or Dedupe?
 
10gb is a waste, if you must connect multiple machines together at faster than 1gb speeds, get infiniband for much less and better performance.

to handle parity, get an expander and an LSI 9266 or one of the higher end areca's. if you're not going to use the raid offered by the controller, and will instead software raid, then spend more on the processor and use your software raid. you won't need more than 24gb of ram.

get a 24 drive case. don't want to get a 24 drive case? then get 2 12s, or 3 8s. get 4tb drives. plug them in. pick your raid software and choose either a higher end raid controller or a higher end cpu to handle it. either will work in windows or linux or freebsd fine.
 
well if you build it yourself, and take that risk, definatley look into some sort of setup for failover and the like. If you're going to run hardware raid then you don't really need ZFS, I'm running a hardware raid for a few pools of disks in raid50 and then using ZFS for it's own benefits and as a way to add SSDs to cache it. ZFS has it's own overhead and advantages and disadvantages. From an administration standpoint I like having the software hybrid (hardware and software) but it's not for everyone

10-Gig is going to be expensive. Switches cost something like 2-4K. Having the back end support 10gig is helpful if you're having a lot of users all pulling from the same server but from what it sounds like you aren't but perhaps in the future you might. I'm thinking something like - File server > 10gig switch > 1000mbit switches > clients

in theory a fat pipe at the fileserver end will allow it to supply more requests and at faster speeds than it being connected at 1gbit

Hyper-V and storage pools might be something to look into - especially the features that advertise speeding up SMB shares but that requires fancy network controllers and such

It all gets complicated but can net you some good performance improvements. I think the metric you need to consider is how will you be using this? as merely a storage, so mostly writes and not many reads? Only large files? That affects how you design the storage.

Yes, dedupe and compression don't make sense on pools that don't use a lot of files, and already have compressed files like jpegs or movies, but I could see it being potentially helpful on lossless stuff like TIFFs etc
 
Dell will be out of your budget in a heartbeat

MD1200 with 12x4TB nl sas drives will be a little over 13k....that's only 48tb raw.
 
I think you could do some sort of failover setup and make almost 2 of everything, and do it yourself, and save the cash.

something along those lines would be worth looking into.

Just saw the single chassis bit in the OP... You could do it with a raid50 setup in a single norco 4224 and set it up with a NAS having the chassis hold the drives as well as the motherboard, ram, and controllers

that's a rather tidy solution but also a single point of failure should a PSU go out or something
 
Yeah I think there is no way he can go Dell unless he loosens his pocket strings

That being said, I might do a fat Norco build for him with knowledge gained from this thread. I'm trying to decide if I should go single chassis or split it into two. If I do a single Norco 4224 w/ 24 bays I can stuff 96 TB Drives in there and do a simple raid6 w/ hot spare setup.

I like the idea of ordering it from a place like AcmeMicro but i'd need one near Vancouver BC Canada if anyone has any ideas please let me know, appreciating all the help.. this should be fun heh
 
Redundant PSU #1...and going to want a real good one for that many drives.

Hot Spares #2 - i would say 2 hot live with 2-3 offline ready just incase.

backup for the data #3.. ? ? ?

going to be tight on $15k..

how important is this data if this server goes up in smoke?

If the mobo dies one day..

et cetera.........

can it have any down time?
 
MrGuvernment said:
Redundant PSU #1...and going to want a real good one for that many drives.

Hot Spares #2 - i would say 2 hot live with 2-3 offline ready just incase.

backup for the data #3.. ? ? ?

going to be tight on $15k..

how important is this data if this server goes up in smoke?

If the mobo dies one day..

et cetera.........

can it have any down time?
Click to expand...

Agree on all counts, He has 15 2TB USB drives (where his current production data is lol) So I can convert those into backup drives.

If I do the single chassis i'd probably do a raid60 with 4TB drives and have plenty of hot/cold spares. But yeah if the raid controller, cpu, mobo ect,. goes tits up... ruh roh
 
Lol usb drives? that stuff should be on enterprise grade tape
 
gigatexal said:
Lol usb drives? that stuff should be on enterprise grade tape
Click to expand...

Indeed! and I happen to have a TL2000 Tape Library in my garage so i'll probably use that. I think the software to manage it is really expensive however, Commvault/Backupexec etc,.
 
Guldan said:
He doesn't have an IT team as it's a small company. Which is why i'm leaning towards just getting brand new Dell stuff but that would put him way over $15k. Also I prefer ZFS but am leaning towards Windows as it is what he knows and easier to manage by non-IT people.
Click to expand...
I tend to think getting non-IT people to manage IT infrastructure is a recipe for disaster.
Guldan said:
One thing you did bring up which i've been considering is 10Gig connections, do you think this is required? I'm just worried it will cost a lot of $$ and not be utilized fully.
Click to expand...
10 gigabit is probably overkill. What kind of files are these? Video, images, virtual machine hard drives, something else? How many people use the files at a time?
Guldan said:
My work file server = (perc6e->md1000/md1000/md1000) at work and the server powering it has 12GB of ram (It's a cheap dell w/ a perc6e raid card). Wouldn't I only need large amounts of ram for Software raid and/or Dedupe?
Click to expand...
Dedup is a waste of effort in 90% of cases. The only time you'll get a savings out of it is when you're storing the same files in multiple places, and that's a bad idea anyways.
Guldan said:
That being said, I might do a fat Norco build for him with knowledge gained from this thread. I'm trying to decide if I should go single chassis or split it into two. If I do a single Norco 4224 w/ 24 bays I can stuff 96 TB Drives in there and do a simple raid6 w/ hot spare setup.
Click to expand...
You'll want to do raid 60 with smaller disk sets to keep rebuild times down.
Guldan said:
I like the idea of ordering it from a place like AcmeMicro but i'd need one near Vancouver BC Canada if anyone has any ideas please let me know, appreciating all the help.. this should be fun heh
Click to expand...
Acmemicro ships to Canada:
Canada Sales:
[email protected]
1-(800) 313-2263
Click to expand...
Guldan said:
Indeed! and I happen to have a TL2000 Tape Library in my garage so i'll probably use that. I think the software to manage it is really expensive however, Commvault/Backupexec etc,.
Click to expand...
Amanda seems to support the TL2000. You won't have commercial support setting it up, but it'll do the job.
 
I also really like the acme micro products, rebranded supermicro stuff. I've had pretty good luck with equipment from them as well.

I'd say 70TB is a lot of eggs in one basket. Managing that much data in one "chunk" is difficult to backup, lots of time. I understand most of it is static but it still sounds like a nightmare to me.

I just provisioned up a 8TB server into half a dozen smaller 1-2TB servers. I'm having much better luck and feel more confident about my backups working well (veeam backups of the virtual machines + file level backups).

I know that data stores of this size are impractical for what you're doing, but maybe chucking it up into nice sized pieces might do you a lot of good. With that much data you really need a solid plan to keep cycling in and out work of whats active and whats not active. I highly doubt that all of that data is accessed at any one time, and trying to keep all of it online in the same spot sounds really difficult, especially in a disaster situation.

Disk-to-disk backup for that much data seems impractical so you almost have to go to tape, and its going to get expensive.

It feels like you're going to war with a truck with a gun mounted in the back instead of a tank.
 
unhappy_mage said:
I tend to think getting non-IT people to manage IT infrastructure is a recipe for disaster.
Click to expand...

Agreed, his current consultant isn't good enough hense him hiring me but I can't provide support afterwards because i'm 9-5 and on call half the time. He'll just have to find someone else when it hits the fan.

unhappy_mage said:
10 gigabit is probably overkill. What kind of files are these? Video, images, virtual machine hard drives, something else? How many people use the files at a time?
Click to expand...

4-5 people at a time maximum, just large images from what i've been told. Currently he is using those 15 USB drives and he says their speed is fine so just about anything is an improvement. Regular gigabit is fine

unhappy_mage said:
Dedup is a waste of effort in 90% of cases. The only time you'll get a savings out of it is when you're storing the same files in multiple places, and that's a bad idea anyways.
Click to expand...

Agreed

unhappy_mage said:
You'll want to do raid 60 with smaller disk sets to keep rebuild times down.
Click to expand...

Good point

unhappy_mage said:
Acmemicro ships to Canada:
Click to expand...

Hopefully the shipping isn't too bad but ill spec something out from them. Appreciate the link, Will make this venture a lot easier not having to get all the parts individually.

unhappy_mage said:
Amanda seems to support the TL2000. You won't have commercial support setting it up, but it'll do the job.
Click to expand...

Yeah I think that's the theme here, no commercial support, I let him know why it's a bad idea. Apparently in 6 months he'll have a lot more $$ to upgrade
 
Guldan said:
Agreed, his current consultant isn't good enough hense him hiring me but I can't provide support afterwards because i'm 9-5 and on call half the time. He'll just have to find someone else when it hits the fan.
Click to expand...

:eek::eek::eek:
If you end up tackling this project, I sincerely hope that you document the hell out of it, otherwise, the next guy in there is what we like to call "fucked".
 
I think you can do it. You should read up on raid and read up on proper fault tolerance practices, then design it, crowd source it (that's where we come in and give you feedback), price it -- see how price comes after a properly designed setup, and then go and build it, OR -

give it to an OEM like Dell or HP and see if they will match your build out characteristics and price point, these days, and in this economy companies are dying for new buisness. They might give you a sweet deal.
 
Guldan said:
Agreed, his current consultant isn't good enough hense him hiring me but I can't provide support afterwards because i'm 9-5 and on call half the time. He'll just have to find someone else when it hits the fan.
Click to expand...
Well, I guess that's his problem, but make sure you make clear in your contract with him what your responsibilities are. Also, you have a contract with him, right?
Guldan said:
Yeah I think that's the theme here, no commercial support, I let him know why it's a bad idea. Apparently in 6 months he'll have a lot more $$ to upgrade
Click to expand...
So why not limp along with the external drives for 6 months, hire some staff or a consultant, and get the right equipment the first time around? The Acmemicro box with FreeNAS or a Solaris derivative on top will be a lovely box, if there's someone around to support it. If he's cut adrift on the sea of command-line interfaces with nobody to help, he's not going to be very happy with the solution or you.
Nate7311 said:
:eek::eek::eek:
If you end up tackling this project, I sincerely hope that you document the hell out of it, otherwise, the next guy in there is what we like to call "fucked".
Click to expand...
Agreed. Documentation on what parts you're buying and why, OS configuration changes, how to do a RAID rebuild, all of that should be part of your deliverables.
 
+1 For documentation, as a sysadmin I share the frustration of troubleshooting something blind. I haven't agreed to anything and I won't unless I am confident I can do the job and know that it won't effect my current job.

If he goes Dell I can talk to my rep and probably save him a fat stack of cash right off the top, they definitely have wiggle room on pricing.

also i've learned a lot in the last few days researching the Norco/Whitebox route as well, maybe it's not for him but good to know such options exist.
 
kudos, keep us informed. Im interested in how you go about this, either oem or diy.
 
If you decide to roll your own stuff be sure and have hot spares or r6. The rebuild time on 3TB drives is going to be massive. I don't even think there is an enterpise level 3TB drive so the chances of another failure during rebuild is high. IMHO 70TB is way beyond throwing together a solution from newegg if data loss is any concern.
 
You must log in or register to reply here.
Back
Top