Build-Log: 100TB Home Media Server

Very nice setup! Can we ask what you do for a living? I noticed that you were able to assemble this where you work.
 
@OP

100TB eh? Yeah, back ups might be an issue :p Seriously though, the only thing that has a chance of backing up something that size is one of those wardrobe-sized tape libraries, controlled by Tivoli or something...unless you went and built an identical server and have all of its drives available as an iSCSI mirror or something - bonded GigE, the whole works.

As for the in-server redundancy...hmmmm....48 drives....I reckon 2 24-drive RAID 6 arrays like you suggested , and lose 8TB to parity. The number of drives per array will depend on what the Areca controller processor can cope with, which I've no idea about.

As an added precaution, I'd set up an account with an SMS gateway like Click-A-Tell, so you'll know immediately if one of the drives fails. E-mail tends to be kinda silent...
 
In a system like this, you *always* put hotspares. It makes immediate notification more of a "hey it's time to do an RMA" notification rather than a "time to run and put a drive in" warning.

Also, I have my Areca card set up to send an e-mail address to my e-mail -> txt converter @ att. (Your phone number @providersmsgateway.net usually).
 
Does the ARECA software support SMTP authentication? I ask because virtually every SMTP gateway provider has this requirement - gone are the days when they simply checked your IP address...
 
Awesome build man! I also would have suggested an SM board but one important thing! Don't forget your areca BBU! That thing is quite important! Also definitely go 2xRaid6 rather than 3xRaid5
 
pjkenned said:
In a system like this, you *always* put hotspares. It makes immediate notification more of a "hey it's time to do an RMA" notification rather than a "time to run and put a drive in" warning.

Also, I have my Areca card set up to send an e-mail address to my e-mail -> txt converter @ att. (Your phone number @providersmsgateway.net usually).
Click to expand...

I don't agree with this. I agree that having coldspares is good (having a drive around so you can make an immediate replacement) but if the machine at your home then you will always be able to replace the drive in <24 hours. I do agree it is important to have a spare so you can replace the disk in a timely fashion.
 
Apologies for the dumbness, but what's an SM board? (I don't get to play with servers and RAID controllers much)

I don't agree with this. I agree that having coldspares is good (having a drive around so you can make an immediate replacement) but if the machine at your home then you will always be able to replace the drive in <24 hours. I do agree it is important to have a spare so you can replace the disk in a timely fashion.
Click to expand...

I suppose there's arguments back and forth. Hot spares means sacrificing capacity, but at least you'll know if they are working. Cold spares are server-external, but unless you fire them up every so often, you won't know if they'll work when you really need them. Good thing about RAID 6 is the likelihood of double parity failure is drastically reduced over RAID 5, so if one does let go, you should have enough time to replace the failed unit.

As far as I know, hot spares are traditionally used for situations where a system administrator might not be around all that often to deal with a disk failure, especially at far-flung installations. Or am I wrong?
 
Last edited:
parityOCP said:
Does the ARECA software support SMTP authentication? I ask because virtually every SMTP gateway provider has this requirement - gone are the days when they simply checked your IP address...
Click to expand...

its not even necessarily software per say as the areca has its own 10/100 port so it can send mails even with no software installed or the machine is MID POST. To answer your question yes it supports SMTP authentication and takes a username/password for the SMTP server but doesn't to my knowledge support encryption like TLS.
 
parityOCP said:
Apologies for the dumbness, but what's an SM board? (I don't get to play with servers and RAID controllers much)
Click to expand...

Acronym for SuperMicro. I used to use regular brands myself until I started working in a data center and realized how superior they are to other brands.
 
parityOCP said:
Apologies for the dumbness, but what's an SM board? (I don't get to play with servers and RAID controllers much)
Click to expand...

SuperMicro motherboard... the X8SIL-F is a current favorite because it is inexpensive, works with low end CPUs, uses ECC RAM, and has remote KVM/IPMI (remote KB/Mouse with a web interface).
 
parityOCP said:
So how would you compare them to a brand like Tyan?
Click to expand...
Supermicro is above Tyan. Best of luck to you if you ever have to RMA something with Tyan. Supermicro is the brand to get when it comes to server boards.
 
Blue Fox said:
Supermicro is above Tyan. Best of luck to you if you ever have to RMA something with Tyan. Supermicro is the brand to get when it comes to server boards.
Click to expand...

Agree here. I had an extremely bad Tyan experience... never again. SuperMicro is top notch... I even had one of their support engineers calling me after hours on a pre-sales call just to make sure I was getting the right SuperServer barebones and UIO card. You usually don't get that sort of thing unless you buy Dell or IBM or HP.
 
I am still in shock and awe of this build log. Just the immense possibilities are staggering.

Glad that it is up and running, but like any new toy, don't stop playing with it till you get it just right!
 
Well i have had a very bad SM experience (3 of em) lately and have since bought some ASUS boards that have been great.
 
Blue Fox said:
Supermicro is above Tyan. Best of luck to you if you ever have to RMA something with Tyan. Supermicro is the brand to get when it comes to server boards.
Click to expand...

It may depend on your area. Here in Europe both brands offer decent tech support by phone with no queueing and turn around on RMAs is about the same. We stock quite a few racks at work and have not really seen any noticeable difference in failure rates between the two brands. Both brands OEM for the Tier 1s, Tyan supply the majority of HP XW workstation boards.
 
Personally I've heard nothing but horror stories when dealing with Tyan in North America. Supermicro on the other hand, I've had excellent experience with. I've spoken with their engineers on numerous occasions and they were quite helpful. 24/7 support on their barebones is nice too. I guess Tyan can't be worse than Chenbro though. Their support was the worst. :p
 
Wow, amazing treadstone! And here I thought my 10 TB server was something to behold.

I am curious though what you are doing to avoid the Raid issues many folks are having with the WD 2TB drives? I know WD makes a Raid version of their 2TB drive that one would guess is an optimized (firmware or hardware) version of the drives you have.
 
Sorry for the LONG delay in my response.
Unfortunately, the server is STILL sitting in my garage in the shipping box on the skid... It's been sitting there since April 2nd, so over a month :(

georgexi said:
what are you using for backup? ;)
Click to expand...

Don't need one really as this is just for storing Blu-ray movies of which I have the original disc for, so if something happens to a drive, I'll just replace the drive and have to backup the movies again...

Krobar said:
Any benchmarks yet?

Are you going for hardware raid?
Click to expand...

I had done some benchmarks on the drives when I flashed them with different firmware versions to see if it made any difference. I may post the screen captures if I find the time for it.

About the RAID, that is something I am going to post about shortly as I would like to get some feedback from others on this subject in particular to my requirements/setup.

iakovl said:
Small, isn't it
Click to expand...

VERY :D

smidley said:
Very nice setup! Can we ask what you do for a living? I noticed that you were able to assemble this where you work.
Click to expand...

I design telecommunication systems. I'm a hardware designer by trade. And it helps that I am my own boss :)

parityOCP said:
@OP

100TB eh? Yeah, back ups might be an issue :p Seriously though, the only thing that has a chance of backing up something that size is one of those wardrobe-sized tape libraries, controlled by Tivoli or something...unless you went and built an identical server and have all of its drives available as an iSCSI mirror or something - bonded GigE, the whole works.

As for the in-server redundancy...hmmmm....48 drives....I reckon 2 24-drive RAID 6 arrays like you suggested , and lose 8TB to parity. The number of drives per array will depend on what the Areca controller processor can cope with, which I've no idea about.

As an added precaution, I'd set up an account with an SMS gateway like Click-A-Tell, so you'll know immediately if one of the drives fails. E-mail tends to be kinda silent...
Click to expand...

Backup: See my comments above.

RAID: I have setup the system as 3 x RAID 5 systems with 16 drives each, then changed the configuration to 2 x RAID 6 with 24 drives each. That was before I moved it into my garage. I only had those setups for a few hours each just to see what the power draw would be like with the different setups. Again, I am going to post more specific information/questions about this shortly.

pjkenned said:
In a system like this, you *always* put hotspares. It makes immediate notification more of a "hey it's time to do an RMA" notification rather than a "time to run and put a drive in" warning.

Also, I have my Areca card set up to send an e-mail address to my e-mail -> txt converter @ att. (Your phone number @providersmsgateway.net usually).
Click to expand...

I bought two extra/spare 2TB drives just in case one of the other 50 drives fail. Since this is more or less just a backup of stuff I can replace somewhat easily, I don't think I will go the hot spare route.

houkouonchi said:
Awesome build man! I also would have suggested an SM board but one important thing! Don't forget your areca BBU! That thing is quite important! Also definitely go 2xRaid6 rather than 3xRaid5
Click to expand...

Thanks houkouonchi, I have looked at your builds in the past and enjoyed reading about them, so it's nice to see you like my little contribution too ;)

As I said before, I had looked at SM motherboards, I just couldn't find one that had the right combination of PCI-E slots...
I was considering the BBU module, however the server will be hooked up to my 5kVA UPS that has a run-time of over 6h at full load and if there is an issue with the UPS, it messages the network/server to shut down. So I think I should be covered...

Wyluliraven said:
I am still in shock and awe of this build log. Just the immense possibilities are staggering.

Glad that it is up and running, but like any new toy, don't stop playing with it till you get it just right!
Click to expand...

Unfortunately, it's not up and running yet...

0xygenthief said:
Wow, amazing treadstone! And here I thought my 10 TB server was something to behold.

I am curious though what you are doing to avoid the Raid issues many folks are having with the WD 2TB drives? I know WD makes a Raid version of their 2TB drive that one would guess is an optimized (firmware or hardware) version of the drives you have.
Click to expand...

I have a bunch of smaller servers that I am currently using that will be replaced by this system. Basically I am trying to consolidate everything into one box.

I upgraded the firmware of the 2TB drives to a different version that is used in the RAID versions :)

criccio said:
Your lack of updates is disturbing.
Click to expand...

Sorry, I dislocated my shoulder just over 3 weeks ago and I'm finally getting back into shape where I can actually do or use this arm again. I still can't lift anything heavy with it, but at least I can move it around without it hurting too much anymore.

Shockey said:
He hurt his arm so i don't think he can move it into place to do anything with it.
Click to expand...

Thanks... It's been a PITA not to mention REALLY frustrating not be able to move the server and start playing with it. Hopefully by this weekend, I'll get this server moved and setup so that I can finally start using the damn thing :)
 
awesome setup! I'm curious to see Speedfan on that baby controlling the majority of the fans...

Also something to think about is the eventual storage upgrade you will want to do. Although 100TB is a huge leap in storage, you must figure that for you to need 100TB and counting that in the near future you will need to upgrade those hard drives. Do you think that down the line when the 3 or 4tb drives come to the surface you would upgrade the entire 24 disc array all at once, or will you prefer to slowly change them out? If you were to change out an entire array you would need to put the data somewhere while you upgrade (unless you like ripping hundreds of discs)

As noted in your previous answers you haven't chosen one system yet, but I would seriously consider putting the system disc into a RAID mirror and having the rest in a storage pool for easy upgrades using WHS V1 or V2 when it becomes available.
 
Even though the all of the fans have a TACH output to measure the fan's rpm, unfortunately none of them are connected to anything that the CPU could access this information from. I am toying with the idea of building a custom made fan controller for this system.

I think 100TB will do fine for a little while. I am not sure if I would upgrade the system if larger drives would come along. What I would consider is to build a custom designed drawer style server system for 2.5" HDDs. Currently the largest size I believe is 1TB for the 2.5" HDDs. Once those get into the 2TB range that would be an interesting little project :)
 
treadstone said:
Even though the all of the fans have a TACH output to measure the fan's rpm, unfortunately none of them are connected to anything that the CPU could access this information from. I am toying with the idea of building a custom made fan controller for this system.

I think 100TB will do fine for a little while. I am not sure if I would upgrade the system if larger drives would come along. What I would consider is to build a custom designed drawer style server system for 2.5" HDDs. Currently the largest size I believe is 1TB for the 2.5" HDDs. Once those get into the 2TB range that would be an interesting little project :)
Click to expand...

Oh interesting on the fan controls....if you're savvy enough to make a custom fan control, then i say give it a shot! That would be a mighty undertaking...however I bet your ears would thank you! definitely interested in the custom drawer for HDDs, i've toyed with the idea of what i'll do when my measly 6TB system runs outta space. Considering the server could hold only 13 drives in total, I figure i'd use an extra case I have laying around and convert it. It's a piece of crap that I got for free; I could load up maybe 3 or 4 shelves running lengthwise in it and by my estimates fit approx 12 3.5 drives per shelf. Not bad considering all I would need is alittle plexi-glass, metal and backplanes!
 
Last edited:
Matt897 said:
Oh interesting on the fan controls....if you're savvy enough to make a custom fan control, then i say give it a shot! That would be a mighty undertaking...however I bet your ears would thank you! definitely interested in the custom drawer for HDDs, i've toyed with the idea of what i'll do when my measly 6TB system runs outta space. Considering the server could hold only 13 drives in total, I figure i'd use an extra case I have laying around and convert it. It's a piece of crap that I got for free; I could load up maybe 3 or 4 shelves running lengthwise in it and by my estimates fit approx 12 3.5 drives per shelf. Not bad considering all I would need is alittle plexi-glass, metal and backplanes!
Click to expand...

I actually drew up a schematic for a distributed fan controller already. The entire system would be able to handle 16 fans and have a USB host interface. I even threw in current measuring ability for each fan to figure out the power consumption/efficiency. It also has a temperature sensor per fan controller. Each fan controller would basically consist out of a small PCB that you would mount close to each fan to measure the temperature as well as control the fan. The design is based on or intended to be used with PWM controllable (e.g. 4pin) fans. I would just need to get my brother to write me an interface for the PC (e.g. a program that can run as a service in the background on an OS like Windows Server 2008 R2 or something similar).

Linezilla said:
Did the op already post the cost of this set-up? Wowza!!!
Click to expand...

It's in a previous post.

sjbwiii said:
More info please. This is for the WD20EADS. Would you, or anyone know if it's the same for the WD20EARS.

Thanks.
Click to expand...

WD20EADS correct. The hardware is identical to the RAID drives. The firmware is what differentiates them. I don't know if this would also work on the WD20EARS. Sorry, don't know, didn't ask. But at least now all the drives support APM!
 
nice build!


http://www.supermicro.com/xeon_3400/Motherboard/X8SIA.cfm?IPMI=Y comes with ombord gfx card.

05052010238.jpg


Did buy this MB for my server. Lots of PCI-E slots
 
agrophel said:
nice build!


http://www.supermicro.com/xeon_3400/Motherboard/X8SIA.cfm?IPMI=Y comes with ombord gfx card.

Did buy this MB for my server. Lots of PCI-E slots
Click to expand...

I was looking at either the X8ST3-F or X8STE, don't remember exactly which one. The problem with the SM server boards is that the build in graphics don't support HDMI out. Also, I wanted to use the i7-860 processor...

Anyway, for now the motherboard I have in there works. If I need to change something on the server, I may consider SM again, but only if they come out with a server board that has the features I was looking for...
 
Your setup makes me feel like I'm playing with misshapen pieces of wood over here.

Very impressive and very out of my realm.


FTP login and pass? :D
 
treadstone said:
I actually drew up a schematic for a distributed fan controller already. The entire system would be able to handle 16 fans and have a USB host interface. I even threw in current measuring ability for each fan to figure out the power consumption/efficiency. It also has a temperature sensor per fan controller. Each fan controller would basically consist out of a small PCB that you would mount close to each fan to measure the temperature as well as control the fan. The design is based on or intended to be used with PWM controllable (e.g. 4pin) fans. I would just need to get my brother to write me an interface for the PC (e.g. a program that can run as a service in the background on an OS like Windows Server 2008 R2 or something similar).

Wow, i couldn't even fathom designing my own PWM board. I'm sure other people have said this, but you....probably could sell alot of those boards here and make some good side dough!

if you could have him write software that could be installed on a client machine that would be really sweet as you would be able to monitor the basic operation sitting at your desk without having to remote access into it
Click to expand...
 
treadstone said:
Sorry for the LONG
Thanks houkouonchi, I have looked at your builds in the past and enjoyed reading about them, so it's nice to see you like my little contribution too ;)
Click to expand...

Trust me, get the BBU. I manage servers where I work and we have over 500 with hardware raid controllers. I have seen corruption issues from not having a BBU or it just flaking out on LSI controllers and its never been due to power loss on the machine but because the machine crashes and has to be power-cycled.

A UPS will not protect you from losing cache data if your machine crashes for some reason or something else happens that you did not expect and I highly suggest getting the BBU as it is well worth it!

I honestly can't say I want to see another one of those fucking areca BBUs as one of the companies we bought servers from didn't include them so I had to install them in about 60 servers and they were all low profile which made it an extra pain int he ass:





 
100tb for bluray?

do you have like $50000 worth of bluray discs?

cool none the less, but going against everything that is [h]ard, this is overkill
 
This is the biggest media server on [H], right? This is beyond impressive!

What do you use for the frontend software? I'd also be interested to see what OS you finally end up with.
 
"upgraded the firmware of the 2TB drives to a different version that is used in the RAID versions"

Can you elaborate on this further? How did you get this firmware, I thought WD likes to keep this secret?

Does that mean your WD20EADS is recognized as a WD2002FYPS or something like that? Can you do TLER on it?

are you willing to share ;)
 
hellosky said:
"upgraded the firmware of the 2TB drives to a different version that is used in the RAID versions"

Can you elaborate on this further? How did you get this firmware, I thought WD likes to keep this secret?

Does that mean your WD20EADS is recognized as a WD2002FYPS or something like that? Can you do TLER on it?

are you willing to share ;)
Click to expand...

Model number and other parameters are stored separately from the firmware. Which means even after you change the firmware the settings including the model number, hardware revision, etc all are kept intact and do not change. The drives are still being recognized as WD20EADS. Sorry, even though I would love to give you the firmware, I can't (not allowed to do so). I know that WD will not and does not hand out firmware unless there is an absolute need for it (e.g. the drive does not perform as advertised). I highly doubt that you will get anything from them even if you'd ask them nicely. TLER can be enabled after the firmware upgrade. I enabled TLER and disabled WDIDLE3 before the firmware upgrade. After the upgrade I had to disable WDIDLE3 again but TLER was still enabled.
 
Awesome build, but could you make those pictures any smaller? Sheesh :p
 
You must log in or register to reply here.
Back
Top