The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
Well, I got my solaris box all up and running properly and semi stress tested and finally bit the bullet to re-allocate my array giving a bit more space now for windows and my linux OS slice:

Code: 
root@sabayonx86-64: 09:18 PM :~# cli64 vsf info
 # Name             Raid# Level   Capacity Ch/Id/Lun  State
===============================================================================
 1 WINDOWS VOLUME     1   Raid6    129.0GB 00/00/00   Normal
 2 MAC VOLUME         1   Raid6     30.0GB 00/00/01   Normal
 3 LINUX VOLUME       1   Raid6    129.0GB 00/00/02   Normal
 4 DATA VOLUME        1   Raid6   35712.0GB 00/00/03   Normal
===============================================================================
GuiErrMsg<0x00>: Success.


I also was able to get everything copied over and then copied back to my machine so I am back to my machine's full glory:

Code: 
root@sabayonx86-64: 09:17 PM :~# df -H
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sdc1              129G    47G    83G  37% /
udev                    13G   263k    13G   1% /dev
none                   2.1M   324k   1.8M  16% /lib/rcscripts/init.d
/dev/sda1               84G    60G    25G  71% /winxp
tmpfs                   13G      0    13G   0% /dev/shm
/dev/sdd1               36T    15T    22T  41% /data
osol:/data              18T    15T   3.0T  83% /osol

Here is a pic of the SAS expander, motherboard and layout inside the solaris box and the pic of the two machines together (60TB total inside the two machines):













I am a couple weeks from moving and when I do I will have a full 45U rack to have all my stuff in. I will take a picture when all is said and done and update my storage totals. I should be at just under 90TB when all is said and done.
 
MatDef said:
Total Advertised: 17TB
Total Available: 15.46TB

Specs:

Shuttle SG31G2
Intel C2D e8500
2gb (2x1gb) ddr2 800 of Crucial Ballistix Tracers
1x1tb Seagate Barracuda hd
2x1tb WD GP
7x2tb WD GP
2xSans Digital TR4M-B
LG GGC-H20L Bluray/HD-DVD drive

Here's an older pic, the only change is another TR4M-B added off to the side. :p

http://imgur.com/twAEm.jpg

And it's running WHS, which I've been very happy with.

http://imgur.com/w6G2G.jpg

Kinda wimpy compared to most here, but I'm glad I finally made the club. :D
Click to expand...

Added a few more drives:

HOmJ7.png
 
houkouonchi said:
Click to expand...

houkouonchi ... i see you're powering both dual-molex connections in the norco backplane ...

does the 850hx have all the needed molex power cables or you had to build/purchase additional ones ?

i have a 750hx, and it doesn't have enough molex connections to power both dual molex in my norco 4220 backplane.

thanks in advance
 
lboregard said:
houkouonchi ... i see you're powering both dual-molex connections in the norco backplane ...

does the 850hx have all the needed molex power cables or you had to build/purchase additional ones ?

i have a 750hx, and it doesn't have enough molex connections to power both dual molex in my norco 4220 backplane.

thanks in advance
Click to expand...

Yes it does. More molex connectors is one of the major reasons I went with the 850hx instead of the 750hx. The 750hx has 8 and the 850hx has 12 molex connectors. My solaris box which idles at around 250w vs 350w (my main box) really didn't need the 850 watt PSU so I literally got it only to avoid using Y adapters.

I did use one Y adapter but the only reason I used it was to be able to reach the boot disk as I don't think the ones that came with would reach it with the other things I had hooked up to them.
 
You don't need to plug in both connectors on each of the backplanes. Just one will suffice.
 
You lot have some more competition now. I'm well into the 100TB+ category now (at least until I decide what to do with these things). I think a few of you will recognize what these things are. :D

 
I'll probably wind up selling them. I like having everything on my Areca card and they're not as quiet as my Norco 4020s. Redundant power supplies and fans are nice though. :p
 
Pretty soon we will see a new thread "Storage Showoff thread - Post your 100TB + systems"

wow
 
One more picture...this goes along with the stack of Infortrend units I posted earlier.

DSC01265s.jpg
 
damn
i think i havnt seen those wd types since the 250g drive age :D

dont think there will be market for 100t+ showoff anytime soon, there are just too few [h]ard|storage fans
 
The ones on the bottom are 250gb WD RE drives, but the rest are larger. Good enough if you need IOPS more than sustained transfers. The Infortrend boxes are rated for about 25k IOPS in RAID 5, so it isn't too bad. I still have another 50+ drives that aren't pictured though. My main file server is ~60tb and I have more scattered around the house.
 
After hurting my back and my car's engine dieing and other things delaying my move in and rack build out I finally got most of the rack setup at my house (but still haven't moved in yet). With any luck I will finish it up and be moved in by this weekend.

I still have to put on the cable management and move my 2U/6TB machine and my main rig (4U/40TB). I also need to buy another shelf and get my laser printer in the rack as well.

Here is a crapy pick of its current state (cell phone):



When its all finished I will finally update my post and I should be up to atleast 90TB.
 
houkouonchi said:
Here is a crapy pick of its current state (cell phone):

http://box.houkouonchi.jp//0724101359-small.jpg

When its all finished I will finally update my post and I should be up to atleast 90TB.
Click to expand...
Three things about that beast:

1) Very tidy cable arrangement, at least for now;

2) AHHHH, an almost-empty full-sized rack! Go away, go away! (curls up in a ball in the corner). :D Fill that thing up, please, it's an eye pain to see all that space going to waste...

3) The 90TB mark IS [H] material indeed. Heck, even 10TB, since most home users don't have that much, but approaching or exceeding the 100TB mark is another thing altogether. Congrats!

Also, I hope your back gets better (you know, 40TB worth of data is A LOT of weight... You should think twice before trying to pick it up without a forklift... lol), and sorry to hear about your car engine :( That sucks.

Now, one more thing: am I the only one missing an Edit button? Where did it go?

[EDIT]Never mind that last part, it appeared again. Last night it was nowhere to be found, though...[/EDIT]
 
Last edited:
PigLover said:
Blue - I thought you'd know enough about managing airflow not to block your HVAC return... :D
Click to expand...
That's in the living room and that piece of furniture literally blocks only a centimeter or so. I do know better than that. :p
 
Holy crap how do some of you afford all of this?

This is my bragging... of how I like to live on the edge. :p

Code: 
[root@borg ~]# df -hl
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             215G  9.1G  195G   5% /
/dev/sda1             190M   25M  156M  14% /boot
tmpfs                 3.8G   48K  3.8G   1% /dev/shm
/dev/md0              1.8T  1.6T  175G  90% /raid1
[root@borg ~]#

/raid1 is my main storage area.

The server has room for 4 more drives, but can't afford to buy any yet, still paying off my furniture lol

My backups actually use like 60% of the disk space. I do all my rotations and such locally, then have a single rotation backed up on separate drives, and an offsite backup job for the most important data, skipping stuff like movies and other replaceable stuff.
 
Red Squirrel said:
Holy crap how do some of you afford all of this?
Click to expand...
I'm not quite sure actually. I work part time and pay for my university tuition out of my own pocket, but I still manage to pull it off...even I'm surprised sometimes. :p
Romano2K said:
Do you know where I can find a new Cooler Master ATCS 201 case? The one used by magnetik in this post. Anywhere in the world.
Click to expand...
Good luck finding one. They've been out of production for nearly a decade now. I can't remember the last time I saw one up for sale anywhere. I would check eBay regularly as that's probably your best shot.
 
Blue Fox said:
You lot have some more competition now. I'm well into the 100TB+ category now (at least until I decide what to do with these things). I think a few of you will recognize what these things are. :D

Click to expand...

damn dude, let me know if you want to sell one, i just got a second one to add to my array :)

newfc.JPG
 
Amount of storage in the following system: 14.5TB Raw, 10.9TB Formated

Case: Fractal Design Define R2 Black Pearl
PSU: Fractal Design Newton R2 650W
Motherboard: Gigabyte GA-H55M-UD2H
CPU: Intel Core i5 650 @ Stock
Cooler: Scythe Ninja 2 Rev.B
RAM: Kingston 8GB DDR3 1333MHZ CL9
GPU (if discrete): Onboard
Controller Cards: Dell Perc 6/i with BBU
Network Card: Intel Gigabit CT Desktop Adapter (EXPI9301CT)
Optical Drives: None
Hard Drives: 8x2TB WD20EADS (Storage) | 1x1TB WD10EADS (Backup) | 1x80GB Intel X25-M G2 80GB (System)
Battery Backup Units: Eaton Powerware 5110 750W APC
Operating System: Windows Server 2008 R2 x64
Fans: 3xScythe Gentle Typhoon 1850 rpm | 2xSilverstone RL4Z S1202512 LIW-3M
Fan Controllers: NZXT Sentry LX
Misc information:
* RAID 6
* Stripe size: 512k
* Cluster size: 64kb
* TLER: Enabled (On all 8xWD20EADS)
* WD GP's Idle time: Disabled

This server is used as a file storage for my various computers/laptops/handheld devices and serves them with various types of files, mainly media ones. Its also a excellent way to learn more about networking and about Windows Server 2008 R2.

server_04.JPG


server_08.JPG


server_03.jpg


server_10_hdtune_02.jpg


server_09_hdtune_01.jpg


Edit list:
* 5/8-2010: Added more information.
* 6/8-2010: Added screenshots from HD Tune.
* 22/8-2010: Updated system information because of new parts installed.
* 16/9-2010: Changed image repository.
 
Last edited:
Storage capacity has been significantly upgraded (doubled, actually) since my previous post at the start of this thread:

Unformatted Capacity: 20.93TB
Formatted Capacity:19.03TB

My goal with this build was a balance of storage density and economy and, other than the inconvenience of replacing drives, it has exceeded my expectations.

Specs are in sig.
Drives (various manufacturers - purchased when more space was needed at lowest possible price/GB):
6 x 2TB
3 x 1.5TB
3 x 1TB
3 x 750GB
1 x 640GB
1 x 40GB SSD

Pics:








WinDirStat:

 
Hi all! Just wanna share with you guys what Ive done:

Norco 4220 w/ 120mm fan board.
Supermicro AOC-USAS-L8i
Chenbro CK12803
Intel Q6600
4GB RAM
Intel S3210SHLC
Corsair HX850
20x2TB Samsung Spinpoint F3EG EcoGreen HDDs

Running Opensolaris. ZFS raidz2


Click to view full size!


Click to view full size!

Code: 
d00dz@TurdBox:~# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
Buffy  36.2T   206K  36.2T     0%  ONLINE  -

Code: 
d00dz@TurdBox:~# zpool status
  pool: Buffy
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        Buffy         ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            c10t4d0   ONLINE       0     0     0
            c10t5d0   ONLINE       0     0     0
            c10t6d0   ONLINE       0     0     0
            c10t7d0   ONLINE       0     0     0
            c10t20d0  ONLINE       0     0     0
            c10t21d0  ONLINE       0     0     0
            c10t22d0  ONLINE       0     0     0
            c10t23d0  ONLINE       0     0     0
            c10t24d0  ONLINE       0     0     0
            c10t25d0  ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            c10t26d0  ONLINE       0     0     0
            c10t27d0  ONLINE       0     0     0
            c10t28d0  ONLINE       0     0     0
            c10t29d0  ONLINE       0     0     0
            c10t30d0  ONLINE       0     0     0
            c10t31d0  ONLINE       0     0     0
            c10t32d0  ONLINE       0     0     0
            c10t33d0  ONLINE       0     0     0
            c10t34d0  ONLINE       0     0     0
            c10t35d0  ONLINE       0     0     0

errors: No known data errors

Code: 
d00dz@TurdBox:~# zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
Buffy                     153K  28.4T  45.4K  /Buffy

36.2TB Avaliable
28.4TB Useable

Shown in a windows share:

Click to view full size!

Did a benchmark of the drives and got:

Write:
16384000000 bytes (16 GB) copied, 51.9885 s, 315 MB/s
Read:
16384000000 bytes (16 GB) copied, 49.4608 s, 331 MB/s
 
kuyaglen said:
almost...

http://i612.photobucket.com/albums/tt203/homo1-2020/20090717_kuyaglen_drives_01.png[IMG][/QUOTE]

Now...

[b]Amount of total storage[/b]:10,750 Gigs
[b]Amount of storage in the following system[/b]:10,013.73 Gigs

[b]Case[/b]: Antec Super Lanboy
[b]PSU[/b]: Antec True Power Trio 550
[b]Motherboard[/b]: Intel DG33TL
[b]CPU[/b]: Intel Q6600
[b]RAM[/b]: 6 gigs (2 gigs Kingston PC2-6400, 4 gigs Melco PC2-5300)
[b]GPU[/b]: BFG 9800GTX
[b]Controller Cards[/b]: Dell AAR-2610SA/64MB
[b]Optical Drives[/b]: generic DVD burner
[b]Hard Drives[/b]: 10x WD 1TB Green Drives / 1x WD 750
[b]Battery Backup Units[/b]:none
[b]Operating System[/b]: Windows 7 Pro x64

This rig is for sharing files at home and at lan parties. No backups planned, important files stored in external offline drives. Having a 9800GTX also makes it a very capable backup gaming rig should the need arise, otherwise the 9800GTX is disconnected and unplugged but still held in place between the back of the case and the bottom drive cage.

[IMG]http://i612.photobucket.com/albums/tt203/homo1-2020/10TB-fileserver.png

10TB-fileserver2.png


10tb-fileserver_04.jpg


10tb-fileserver_01.jpg


10tb-fileserver_03.jpg


10tb-fileserver_02.jpg


10tb-fileserver_05.jpg


10tb-fileserver_06.jpg
Click to expand...
 
Last edited:
If anyone is curious I updated my post to include my new pictures of my rack and systems. My new total is 91.4 TB:

http://hardforum.com/showpost.php?p=1033721843&postcount=7

Here is a quote of the page but I can edit out the quote if it annoys people.

houkouonchi said:
Updated 08/09/10:

Total multiple system storage 91.4TB
Total single/internal system storage 40TB

Rack pics:
Rack from the side (Without the side panel on yet:

Rack from the Front (machines in order: zeroshell, kaizoku, chikan, osol, dekabutsu)

Cables going from the rack to the cloest in the master bedroom which in turn goes to my desk that has my monitor, speakers, mouse, etc..). The 'dekabutsu' machine is also my normal use rig:

Cables that go to different drops throughout the house (through the attic) that go into the patch panel at the top of the rack:

Back of the rack (with some view of the cablemanagement):

Pic of the top half of the rack (including patch panel):

A little bit lower where you can see the monitor/UPS/switch:

A bit lower where you can see the cables to the switch nad digiCM (I havent ran cables for the digiCM yet):

Shot of behind the rack with power usage of all machines turned on and semi-idle (1006W):

Another shot of the front of the rack (with flash):



Main storage box (bottom machine in the rack 'dekabutsu').

Total Storage 40 TB
Usable Storage: 36TB (raid6)

NORCO RPC-4020 Case (4U)
Supermicro MBD-X8DTH-iF-O
Dual Intel Xeon E5530 CPU (2.4 Ghz)
48GB DDR3 PC-1333 ECC Memory
MSI Geforce gtx 470
ARC-1280ML raid controller
20x2TB Hitachi SATA (raid6)
OS: Sabayon linux x64

Oldest pics:

http://box.houkouonchi.jp/newraid/dsc_5338.jpg
http://box.houkouonchi.jp/newraid/dsc_5339.jpg
http://box.houkouonchi.jp/newraid/dsc_5340.jpg
http://box.houkouonchi.jp/newraid/dsc_5330.jpg
http://box.houkouonchi.jp/newraid/dsc_5332.jpg

Old Pics:

















I use this box for my main storage of multimedia. It is primarily comprised of unlicensed anime/asian drama and some other stuff I would rather not say on this forum. I also use this as my gaming rig.

Backup box (labeled 'osol' in rack 2nd from bottom)

Total Storage 20 TB
Usable Storage: 18TB (raidz2)

NORCO RPC-4020 Case(4U)
Supermicro C2SEA-O Motherboard
Core 2 duo E6700 (2.66 Ghz)
4GB DDR3 Memory
AOC-USAS2-L8E SAS2 JBOD controller
20x1TB seagate (raidz2) 18TB useable
Running Open Solaris snv134


Second backup box (labeled 'chikan' in rack 3nd from bottom)

Total Storage 6 TB
5.25TB (usable raid5)

Supermicro SC825TQ-560LPB case (560watt/2U)
ASUS server board (forget the model)
Core 2 quad Q6600 2.4 Ghz
8GB DDR2-800 Memory
8x750 GB Seagate HD
ARC-1220 raid controller

Misc usage server (labeled 'kaizoku' in rack 2nd from top):

Total Storage 3 TB

Supermicro CSE-811T-260B Case (260watt/1U)
Another ASUS server board I dont know the model of
8GB DDR2-800 Memory
Core 2 duo E6600 2.4 Ghz
2x.1.5TB Seagate disks (no raid)

Router box/zeroshell (labeled 'zeroshell' top machine in rack):

Total Storage 3 TB

Supermicro CSE-811T-260B Case (260watt/1U)
Supermicro X7SPA-H Atom D510 board/CPU combo
3GB DDR2 SODIMM
2x.1.5TB Seagate disks (no raid)

This does bonding via VPN of my two 35/35 FIOS connections at home. Here is a picture of the FIOS connections outside the house:

One looks quite aged doesn't it? =P

Covers Open:


New ONT on the left and old one on the right (close ups):



The two BBU's:


Speedtest:
868194348.png


Colo Box (colo'd at work)
Total Storage 13 TB
Usable Storage: 10TB (raid6+raid10+raid0).


I upgraded this system to a new case and mobo so my ARC-1280ML no longer has to be jerry rigged as you can see from the old pics. New specs:

2U supermicro (SC825TQ-R720UB)
This has dual 720 watt 90%+ effeciency PSUs
Intel Xeon E5530 2.4 Ghz
24GB DDR3-1333 Memory
8x1500 GB seagate 7200.11 drives (raid6/raid10)
4x256 GB OCZ Core 2 SSD's (raid0)
ARC-1231ML
Supermicro/Intel Quad Gig UIO NIC (AOC-UG-I4)
Onboard graphics

Total Internet connectivity:
1400 megabits.

New Pics:















Oldpics:

Front:
http://box.houkouonchi.jp/newservdc/front.jpg

Back:
http://box.houkouonchi.jp/newservdc/back.jpg

Some pics when I upgraded it to 8x1.5 TB drives:

http://box.houkouonchi.jp/server_hd_upgrade/DSC_0761.jpg
http://box.houkouonchi.jp/server_hd_upgrade/DSC_0763.jpg


I use this server to host several game-servers, seed torrents, and run the free image hosting I setup for hardforums. It has 400 megabits of internet connectivity via 4 NICs. I also use this as a backup of my important files that are on my home system.

In the future the 4x2.5 inch drives bays will become SSDs (probably in 1-2 months) to help bring down the I/O load on the server from seeding torrents.


Mythbox.

Total Storage: 6.4 TB

Raid: 3.5 TB useable 4 TB total:
Nonraid: 900GB
External 1.5TB

Cooler master CM stacker Case
Core 2 duo E6850 3 Ghz
4GB DDR2-800 Memory
Nvidia Geforce 8600GT
Promise EX8350 Raid Controller
8x500 GB 7200.10 seagate drives (raid5)
3x300 GB PATA maxtor (LVM)
2x750 GB eSATA seagate external drives.



This is the box which i used for all my TV. It has 2 genpix DVB-S tuners for HD sat, 1 nexus DVB-S for non HD sat and 2 pchdtv cards for OTA HD. It can record 5 things at once and get non-respomcressed sat streams from dish network.
Click to expand...
 
Origin_Unknown said:
kuyaglen is that graphics card severely squif in the slot or is it my eyes?
Click to expand...
You are correct it is not plugged into the pci-e slot. I dont have any other smaller cards that are just good enough to game with. To even get the card in I have to remove all the drives in the bottom cage and the two drives screwed into the bottom and then remove the raid card and then unscrew the drive cage from the bottom so that I can bend the cage back and angle the card in/out. So I live it in be cause its fine there being held in place, being squished between the back of the case and the drive cage and its simple to just push it into place when need be.

Think of it as a ghetto version of Nvidia's Optimus technology.
 
Just passed 11TB with my home server.

Atom D510 CPU on Supermicro X7SPA-HF
Geil 2GB memory
Supermicro AOC-SASLP-MV8 PCIE SATA controller card (no raid)
750GB Seagate, older leftovers one of these is OS
5x 2TB Seagate new 5900rpm models
1TB Samsung F1 5400rpm leftover
Yeong Yang cube server case
Seasonic S12-550 80+ PS
Win Home Server

Case has room to hold a good 15 or 16 drives. The motherboard and that controller card will top out at 14 SATA devices total.

The Supermicro X7SPA-HF is a real gem, it has KVM over IP built in and you can even feed it disk/floppy images over the network.
 
to : houkouonchi

Last time I worked with full rack (many years ago), I was told to position the equipments properly for balance and stability consideration.

I saw some of your rack pictures and may I inquire you reasoning for loading all the heavy equipments on the upper part of the rack?

Cheers
 
lightp2 said:
to : houkouonchi

Last time I worked with full rack (many years ago), I was told to position the equipments properly for balance and stability consideration.

I saw some of your rack pictures and may I inquire you reasoning for loading all the heavy equipments on the upper part of the rack?

Cheers
Click to expand...

The laser printer is pretty decently heavy as well but it was either going to be nearly all in the top half or all in the bottom half. Since I wanted the printer to be in the rack and it really needed to be in the bottom for easy access to things I print (the top would not work). Also I wanted to put some of the lighter machines at the very top and wanted them grouped together still. I also had considerations about where I wanted the monitor/keyboard to be.

Generally I didn't see a problem as DAMAC racks are pretty heavy duty. We have tons of them (exact model) at work and I have seen racks that just have the top third loaded with machines (with quite a bit more wait then I have loaded) and I have never seen a problem with bending or warping or anything like that.
 
He means it is top heavy and could fall over if it isn't bolted to the floor.
 
MaxBurn said:
He means it is top heavy and could fall over if it isn't bolted to the floor.
Click to expand...

It is bolted down into the floor. The bolts are several inches into the concrete slab that the house resides on using those bolts you hammer down so they wont come back out (without some major trouble).
 
Status
Not open for further replies.
Back
Top