• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

53TB+ Home Server COPY Help ;D

R

Renner Technology

Weaksauce
Joined
Aug 16, 2008
Messages
118
Maybe to spark some interest and get some help: :D
(without trying to sound like a dick..)

some of what I can/am working with
4556306377_bd045caf89_o.jpg

- All the servers haven been built this week. Minus the openfiler...
- 53TB total ;D
- I keep resetting it up and trying to figure out what I want exactly and what will
work the best! :?

16x computers cranking out movies ;D
4556303687_abe06218f5_o.jpg



==TO GET IT OUT OF THE WAY==
Why so many servers and cluttered garage?
- I am a pack rat that looks every day for good buys and when I see them, buy. I have this no stopping
hunger to mess with anything computer wise. Learn it, Test it, Predict what will come next.
- My shop is my over sized 2 car garage. yes who ever posted that below gets a cookie.
(hopefully no one backs up their truck and runs away with everything) <- actually dreamed about that last week! lol
- I do have a computer company that I run on the side of going to school. But this place is for me to learn and have fun


===THE FOCUS===
Basically my dream is this:
- 6x home theater computers around the house
- xbox, ps3 able to pay movies/music
- every movie stored
- an entire collection of every song in once spot! for the entire fam to enjoy!
- have a double server that backs up the main server!

I know there are tons of threads of how people set there stuff up.
but I am sick of messing with anything and everything....


My question to you is
- what do you use (just good to hear things again after reading posts)
- what would you suggest with trying to use Boxee as my streaming source
- best way to keep all my music organized while combining about 20 sources!!

I really need some help on how to go about sharing all my music and what is the best
way of gathering it into once location

------------------------------------------------------------

As far as hardware..
basically what ever is needed I can dedicate to this project. As one guy below stated perfectly
everything is scattered and not really being used correctly. Its like this big mess of wires
and I just need some help putting it right.



WHS
- I just picked up 2x of them a little while ago for dirt cheap. I just started playing with them last week.
- each have 500g HD
- I plan on keeping one and giving the other to a buddy for his bday


OpenFiler System
-e7300
-4g ddr2
-20 hot sway rack
-4x 1.5tb seagate
-6x 1tb seagate
-1x 80g main
- Intel 1000gt NIC
--- this computer is/has been used to store and back up data.

VMWARE
- quad c2d q6600
- 8g DDR2
- 6x 500g Seagate
- Intel 1000gt NIC
--- server used for testing a lot of dental programs and etc personally
it has been an on-going learning project for me to become better with.

All the other servers I am not going to list the rest of the parts and what not....
-duel socket i7 xeon motherboards with e5506's
-quad c2d's
- 20TB WD Green 2TB Hard Drives
- 15TB WD Black 1TB Hard Drives
- 4x 1.5TB Seagates
- etc...
-2x areca ARC-1280ML-2G PCI Express SATA II Controller Card - 24port
-1x HighPoint RocketRAID 2320 PCI Express x4 SATA II RAID Card
-1x SUPERMICRO AOC-SAT2-MV8 64-bit PCI-X133MHz SATA Controller Card 8port
-1x 3ware 9650SE-4LPML KIT PCI Express Lanes: 4 SATA II Controller Card
 
Last edited:
You will have the dubious honor of being one of the first people whose house will be raided by the RIAA.

That aside, that is one crazy setup. I'm afraid to hear the cost of electricity... :eek:
 
its all legal (sadly enough..)
I am so damn sick of my family buying new DVD's for my little sister!
I just found 4x copys of CAR's!!!!!

Power bill does suck ;/
 
OMG that is amazing. I'm creeping up on 10TB total storage and I feel like it's a huge burden, I can't imagine having that much to keep track of and backed up. I just use mediabrowser/metabrowser and a samba share to my ubuntu server's drive, but that's a little bit too basic for what you have going on there.
 
That is awesome!!!!

How much is your power bill?

You must be using between 6000 and 9000 watts!!!

Maybe $500/mo?
 
Last edited:
I have just about every song ever made (I know some radio station guys), and its only 500 GB in MP3 format.

They have some programs out there that will read the ID3 tag and make files and folders from the artist and album, winamp maybe?
 
Should have got a professional program to encode, they can use GPUs and encode an entire movie in 20-30 min.
 
so is your dad an IT Director or something and bring home extra stuff once in a while or did you guys go out buy all that stuff? I know he's not an admin, we don't make that much. Either way, impressive.

I used to have one of those google machines at my last job, didn't use it that much. It crawled our file shares and indexed reports/etc for company wide searches, are you finding it helpful?

EDIT:
Oh, back on the subject - I don't know what you are asking for.

If you want to mirror servers use DRBD - http://www.drbd.org/ - since you are using openfiler, there is a howto guide for DRBD+OpenFiler - http://www.howtoforge.com/installing-and-configuring-openfiler-with-drbd-and-heartbeat
 
Last edited:
Gunnery Sergeant Hartman said:
Holy Jesus...
What is that?
WHAT... THE FUCK... IS THAT..!?
WHAT IS THAT!?
Click to expand...

I wish I had a basement. Damn Florida and below water level... pfft..
 
Should definitely cover that garage window. Someone could look in there and see a motherload they can back a truck up to without raising much suspicion. What a person cannot see does not exist. Sweet usage of space though, looks decently organized for the amount of gear you have in there.
 
Damn that's an impressive spread, I can only dream about owning hardware like that at the moment :( Good job! :D
 
Ok im really confused by this post.

I take it you have a shit ton of movies, but why so many servers?
That just seems really inefficient and difficult to manage....hence this thread.

If you want to stream to a bunch of devices and have a single place for your data to reside, then you need to do some serious consolidation.

Im taking it the WHS arent housing all this data, and they cant backup all that other shit, so what are they used for? Why 2 of them?

Really need some more information.

Tell me more about the fileservers.
What drives
What RAID, HBA, etc.
What CPU's, how much ram

Should really consider consolidating your efforts, it will save you a lot of headache and electricity.

Also why not run your test environment, openfiler, game server on vmware?
Also what is openfiler used for?
 
1042TB, I don't understand how you can copy that much data with a 53tb system. Or what 1042TB would even look like. Typo I'm assuming? And as odd as it may sound, I think the easiest way to set up the music would be to get itunes, tell it where to copy music to, and just drop folders into the library. It will sort it, and make a searchable catalog for you. After that you can set it up to show you the duplicates and then go through and delete any that are doubled up. Then if you don't like apple or don't like itunes, just delete the program.
 
I don't use WHS, but I do use FreeNAS for my Server. I love the capabilities of it. Right now I've got 2 sets of 4 HD's in Raid5. From there, I just mount the drives on my HTPC and set MCE to scan those folders/drives for movies and music. Done.
 
Renner Technology said:
My question to you is
- what do you use (just good to hear things again after reading posts)
- what would you suggest with trying to use Boxee as my streaming source
- best way to keep all my music organized while combining about 20 sources!!
Click to expand...

For my setup @ home, I use Debian Lenny + Samba. HTPCs access the data via XBMC (which is the core of Boxee). It runs fine but I found out recently that if you have a long copy session from the NAS, smbd goes nuts and uses a lot of CPU. Gotta figure out what causes it.

As nitrobass24 said, why so many servers? Main + backup, that's it. You can make your tests on a platform with a single HD in it.

My suggestion/recommendations would be as follows:

- Forget Microsoft, Openfiler, Freenas.. make your own custom install using a linux distribution (OS + Samba + webmin + SSH + rsync). It will be more flexible.
- Software raid via mdadm (cheap and works extremely well)
- EXT2/3/4 is HORRIBLE for storage! Made that mistake the first time...don't do it, use XFS.
- Get a real GB network card on your servers (not onboard).
- People should only have read access to the shares.
- Make a separate share that is writable for others to upload files. Double check them then move them (If you want consistency, you will unfortunately need to check everything before it goes in)
- Use rsync to move files from PC->NAS and NAS->Backup server.


As for the music organization, you need to think out a system that fits your needs. Make sure to think it through as you probably will need to check your music files every time you make a change. It can be a pain if you have a large collection.

My current solution for music:
- Absolutely no single songs, full albums or nothing.
- EVERY SINGLE file is properly tagged + album art (if exists)
- I keep an organized folder structure for base style: Chiptunes, Classical, Metal, OST, Other, Punk-Alternative, Rap, Rock, Techno, Videos. Techno has sub-folders for other genres: Acid, Drum&Bass, Happy Hardcore, Hardcore, House....
- The sub-genres are inside the ID3V2/FLAC Tag
- Folder name structure: Artist - Year - Album [bitrate/FLAC]
- File name structure: Track - Title

The first sorting from the 20 sources will probably be a pain in the ass. Consolidate all the songs in 1 location, use a mass tagger to rename the files and structure them accordingly. Place every album in a seperate folder. Delete duplicates, then start checking them 1 by 1. I find that Tag&Rename is by far the best tagger.

Best of luck :)
 
great suggestions.

I've used tag&rename with good results before as well
 
HOly crap!


That is for HOME use??? That system is bigger than what we have at my work with ~150 employees.


Almost make me ask what you do for a living to afford that much gear.


*drool*
 
I fully agree with phildabill but want to add 2 things that could also work for you:

Hardware:
- get a good GBit NIC, like mentioned before - best would be an Intel PCIe GBit like the CT for example
- get a good GBit Switch. Without that, the whole network makes not much sense, as Ciscos are maybe a bit too complicated and way too expensive, I could highly recommend the HP ProCurve series. Relatively cheap (8 Port for 150&#8364; here in Germany) and the 24 Port should be around 300&#8364; - should be much cheaper in the US. Silent, manageable (just in case) and works very good.

Software:
- Maybe think about using FreeBSD (8) with ZFS or OpenSolaris. ZFS is way more secure than mdadm, but you need at least 2GB of RAM in this machine. You don't need a RAID controller, ZFS will do that for you and it supports Stripe "RAID0", Mirror "RAID10", Stripe+Mirror "RAID10" over X volumes, RAIDz1 "RAID5", RAIDz2 "RAID6" and also RAIDz3 "RAID7" - means: Parity over 3 disks which could come very helpful if you have a lot of HDs inside the server. Running OpenSolaris would allow you to easily setup a VirtualBox to do stuff like fileshare, music sharing, web hosting etc inside.
- Rgd. music: I can recommend mt-daapd aka Firefly Media Server (http://www.fireflymediaserver.org/). It scans the folders you share automatically, so all you need is to drop the files into the "music" folder and after a few minutes it's inside your iTunes (or WinAMP when you use the DAAP Plugin).

What kind of system do you WANT to run? Windows? Linux? UNIX? Maybe interesting to know ;)

PS. Firefly is also available for Windows ;)
 
GeorgeHR said:
You overwrite some.

But who has the time to watch 1042TB of movies?
Click to expand...

Clearly it is a typo. 1042 TB? That's minimum of 521 harddrives. Unless he copied it back and forth a couple hundred times and counted that, then I could see that adding up pretty quickly.
 
updated the post a little bit, if it is still unclear... I am sorry but trying to lay it all out very
quickly and shortly... my head is swimming trying to make things short and sweet

before the 1054TB was a typo
I meant 1054G's of movies I just backed up

also everything is running on pure GIG switches. Linksys Business 24's. 2x of them like
in the picture.
 
Sounds like you need a front end application like Win 7 media center that is real light weight and like 6 atom ion boxes to run the 6 tv's you have around the house. Media browser is a cool app for mediacenter that allows multiple sources to be indexed in one location. Works great for tv shows and movies. Then all you would need is a real simple setup with a logitech harmony one click everything for around the home.

Of course on the back end you would have some work cut out. For example you could take all of your storage and condense it into 2 places, for example a nice raid 0 share for everything and a raid 10 backup. I guess if you really wanted to you could create a custom database program that can store all of the media tag information from your files, and have a media type tag then either allow real time streaming or some form of trans-coding to the media center boxes. For example if you ripped all of your dvd's to iso's you could setup a stream over the gigabit network to deliver your content in a format the device understood. Maybe even take this a step further and make a cool web-page frontend (uses flash or html5) that could read from the database. With filters for artist genre or even movie category you would probably get close to what I think you are looking for. You could then setup the other boxes to do load balancing on your web-page and even your database.

Sounds like a fun project if only to learn the basics behind everything :)
 
You must log in or register to reply here.
Back
Top