The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
Sorry about that, I just had so much on my plate and getting some of the parts took forever.Had the enclosure and the 52 2TB drives for over a month and finally got the remaining parts just over a week ago.

The server is more or less assembled, however I am still working on changing out a few cables that I'm not happy with. Build log should be up soon (that is if I can find the time to post it).
 
Total - 90.41TB Advertised
Main - 49.048TB Advertised


Main PC is
I7 920
Gigabyte X58AUD3R
12Gb of DDR3 1333 at C7 (pretty good for budget C9 stuff)
ATI 5870
Adaptec 51645 + BBU
1X 128GB G-Skill Falcon SSD
1X 320GB 2.5" 7200rpm
1X 600GB Vraptor
12X 2TB
8X 3TB
2X 30" Dell 3008
CM Stacker ST101
Upsonic 2000va UPS
Logitech M950 and G19
Windows 7 Pro 64bit

My main rig is used for pretty much everything, gaming AV ect

This PC goes to LANs pretty regularly and has been heavily modded. I have added a full sized window, an 8" LCD into that so i can have dual screens at a LAN, the inside has been painted black with custom cables and full cable management and now covered in Carbon fiber

Backup server - 40TB Advertised
E5200
Gigabyte P35 DQ6
4gb 800mhz DDR2
1X 500gb
13X 1.5TB Hotswap
10X 2TB HotSwap
Lian Li A77
Server 2008 R2 Enterprise

My backup server provides redundancy for my main rig. i run a full back up every 6 months and incrementally backup every month or so
in between each backup all new data is duplicated across 2 or more HDD in the main rig

Misc
Acer 1430 11.6"
upgraded to 500gb and 4GB of Ram
Windows 7 Home Premium 64bit

This notebook is my torrent box, it runs 24X7. i use this rather than my backup/main rig due to its low power nature

usb 160gb + 750gb

dsc006121.jpg

dsc006411.jpg
 
Last edited:
Its based on the Backblaze case design but redesign with a lot of improvements, custom case was ~$1200 from protocase. Hot-swap, no port multipliers, all drives/fans have rubber vibration dampeners, 5x 200CFM+ fans, redundant power, SAS support, and forward compatible 6Gb/s SAS/SATA.
 
rsriprac said:
Its based on the Backblaze case design but redesign with a lot of improvements, custom case was ~$1200 from protocase. Hot-swap, no port multipliers, all drives/fans have rubber vibration dampeners, 5x 200CFM+ fans, redundant power, SAS support, and forward compatible 6Gb/s SAS/SATA.
Click to expand...

Cool, I was thinking of doing the same. I also have a local metal shop that can build this for me as they already build all my other enclosures I design for me.

Was going to do a drawer type design but then decided to go with the front access hotswap type drive bay setup. I am still toying with the idea of designing a drawer style drive expander with a custom backplane with integrated SAS expander chip, etc.

What drives are you using for your setup?
 
> I also have a local metal shop that can build this for me as they already build all my other enclosures I design for me.

Thats pretty cool, I bet the pricing is much better the Protocase, I think they are expensive. I think if I need more I'm going to see if I can get ppl in TW to make them.

Currently I'm going to test it with a few Hitachi 1TB 7K1000 I have sitting around, once I can verify there is no compatibility problems. I haven't decided the final HDDs, however it will be 2TB disks.

I only wish I have money for a bulk order of 15K disk to do a stress-test on the rig.


Btw, was it you that does circuit fab for a living? I remember someone mentioning something like that in another thread.
 
Yep that was me. I design telecom gear. From concept, through schematic, layout, mechanical design to the finished products..., whatever it takes.

I assume you have space for 60 drives in your chassis ?

That's based on your statement of 118TB... 120TB/2TB=60 drives less one for your RAID (5?) setup I assume ?
If that's what you were thinking, I think you need to revisit your setup as that would be playing with fire! Also, what controller do you have in mind?
 
wow 1200 for one of those cases custom made? that was a lot more than I thought they would of cost.

But 60 drives is a nice amount.
 
@treadstone

>I assume you have space for 60 drives in your chassis ?

Actually its 3 hosts HDDs and 56 in the array.

>Also, what controller do you have in mind?
So far I've got:
2x ARC-1680ix-24
1x ARC-1680ix-12

waiting on 2x HP SAS Expanders, going to test them to see if there are any performance difference doing ratios 1-lane/1-drive versus 1-lane/~3-drives.

If 1-lane/~3-drives works well, I'm thinking of running ZFS over JBOD'ed array. The 1-lane/1-drive will segment the array, so running RAID-6 the array loses 6 disks, bleh.

Would it be possible for me to PM you a few Qs about some circuit design? I've been thinking of a way to hook up LED outputs but I'm not sure whats the best way to go about it (IC component-wise etc). My electronic skills is limited although I have design some boards and got them made at Olimex (http://olimex.com/). I use to program microcontrollers for fun but its been a while.

@paradoxblue
>wow 1200 for one of those cases custom made?
Unfortunately, yep. I've optimized the design, initially it started at ~$1700 and I got it down to $1200. I haven't had a chance to see what prices oursourcing to Asia will be but very surely it'll be much less.
 
rsriprac, PM away... I'm always happy to help out

What program did you use to design the enclosure?

I use solid edge for my mechanical (sheet metal) designs.

How do you envision your 1-lane/~3-drives to work with the HP SAS expander?
 
rsriprac said:
@paradoxblue
>wow 1200 for one of those cases custom made?
Unfortunately, yep. I've optimized the design, initially it started at ~$1700 and I got it down to $1200. I haven't had a chance to see what prices oursourcing to Asia will be but very surely it'll be much less.
Click to expand...

Any pics of this? you could get 3 or 4 norcos for that cost. lol
 
treadstone is right, space is an issue and the challenge is fun. (=

Heres a teaser:

duHKF.jpg


The shot is kinda old, the screws were place holders but now the correct screws are in place (flush).

@treadstone
>PM away... I'm always happy to help out

Thanks, will do once I can get my notes together.

>What program did you use to design the enclosure?

I use SolidWorks. I'll check out solid edge, looks interesting from the brief looks of it.
 
Last edited:
Neat. I assume that the matrix block of 8x8 LEDs at the bottom of the front is what you are trying to control?

Are any of the holes on the side for slide rails or are they for the grids that hold the HDDs inside the enclosure?

With almost 60 drives this is going to be an EXTREMELY heavy case and you will need at least two people to lift it especially when you are going to try and mount it in a rack :)

The Chenbro RM91250 I have is already heavy on it's own but with all 50 drives mounted in it, it's next to impossible to lift! :)
 
>I assume that the matrix block of 8x8 LEDs at the bottom of the front is what you are trying to control?

Yep, thats what I'll be PM'ing you about, hehe. (=

>Are any of the holes on the side for slide rails or are they for the grids that hold the HDDs inside the enclosure?

The holes are for an internal grid to hold the HDDs and it also provides a great deal of lateral rigidity. The original backblaze pod looks like it will flex a lot when the case is fully loaded.

That Chenbro RM91250 looks crazy!
 
OK I have to ask, how are you cabling that thing? My big reason for not going blackblaze(ish) was that I didn't want the port multipliers and I couldn't find backplanes that would work easily.
 
Amount of storage in the following system: 10.5TB advertised

Fractal Design Define R2 Black Pearl
Corsair HX520
MSI 785G-E53
AMD Athlon II X4 620
3x1GB Corsair Dominator (TR3X3G1600C8D)
Adaptec 1430SA (used only as extra ports, no RAID)
4x WDC WD10EADS-00L5B1, 2x WDC WD20EARS-00S8B1, 1x Seagate ST31000520AS, 1x SAMSUNG HD154UI
Arch Linux

This PC is used just as a file storage for my HTPC, so no need for RAID or backups. I also use it for running downloads durring night.

fractalfront.jpg

fractalback.jpg
 
Amount of total storage - 12TB Raw

Amount of storage in the following system - 12TB Raw 9.19TB Formated
Case - Nocro 4020
PSU - Rocketfish 700W
Motherboard - Cheap Intel P35 Board.
CPU - Q6600 Currently
RAM - 4GB DDR2 800
Controller Cards - areca ARC-1680LP
Optical Drives - None
Hard Drives - 12x500GB WD5000AAKS | 4x1.5TB ST31500341AS
Battery Backup Units - 750W/30min APC
Operating System - Windows Server 2008 R2

My system mostly used for media storage. TV, movies, and Pr0n. I also use it to store drivers, and software media ISO. Its currently about 50% filled I just added the 12 500GB drives this week, since I got them for free from work. They were brand new taken out of PCs that were upgraded to SSDs right away.

I don't have a backup system since 98% of the stuff I have stored was downloaded in the first place.

I am currently in the process of replacing the system board and CPU either with something less power hungry. I'm not sure If I am going to upgrade to a Xeon, and also use it to host my VM servers, or get a low power AMD 240 CPU/board. I will likely add another 3-4 2TB Raid5 when I do this.
 
@QQ~: my system with X4 620 undervolted to 1.12V has a 78W idle power draw when all HDD's are active, 62W when only the system drive is active and the rest is in standby mode.
 
Amount of total storage 16TB
Amount of storage in the following system 16TB

Case - Norco 4020
PSU - antec basiqpower 450w
Motherboard - supermicro MBD-X7SBA
CPU - intel core 2 duo e6700
RAM - 4gb curcial 667 ECC ram
GPU (if discrete) - onboard
Controller Cards (if any) - 2x supermicro AOC-SAT2-MV8
Optical Drives - sony slimline
Hard Drives (include full model number) 7x WD 1TB black(WD1001FALS-OOJ7B), 2x WD 1TB geen(WD10EADS-00L5B1), 3x seagate 500GB(ST3500320AS), 2x seagate 500GB(ST3500630AS), Seagate 1.5TB(ST31500341AS), 2x hitachi 2TB(HDS722020ALA330)
Battery Backup Units (if any) - cyberpower 950w 1500va LCD version
Operating System - WHS

primary use is music/movie/picture storage. also have lots of undergoing programming projects stored on it. added lots of capacity this time around, hoping to get the rest of my DVD collection on it.
couple pics
server.png

100_1820-1.jpg


need a new shot but the lighting in my basement sucks hardcore
 
Last edited:
Amount of total storage: 20.7 TB
Amount of storage in the following system: 20.7TB

Case: Norco 4220
PSU: Seasonic X650 Gold
Motherboard: ASUS M4N72-E
CPU: AMD Phenom II X4 945 Deneb 3.0GHz
RAM: 4x2gb OCZ Platinum DDR 800
GPU: Headless system
Controller Cards: Two Intel SASUC8I
Optical Drives: LITE-ON DS-8A4S slimline dvd burner
Hard Drives:
8x Western Digital 1.5 TB (WD15EARS),
4x Western Digital 1 TB (WD10000CSRTL),
2x Western Digital 750GB (WD7500AADS),
4x Western Digital RE3 500GB (WD5002ABYS),
2x Western Digital 400GB (No model number handy)
1x Western Digital 320GB (No model number handy)
1x OCZ Solid 80GB (discontinued jmicron trashy drive with that stupid freezing problem) as a hybrid cache drive.
Battery Backup Units: APC Smart-UPS 2200VA (SU2200NET)
Operating System: Opensolaris 2009.11

Pictures will be forthcoming as soon as my stupid camera finishes recharging.

Most of the storage is for TV shows, movies, pr0n, and backup copies of all my physical installation media. The smaller drives are used as individual virtualbox drives that allow for each of the people in the house to have a virtual XP installation with a private scratch space for downloading stuff. They also serve as PS3 media server heads, since opensolaris sucks as far as compatibility with linux apps goes.

Network access is via samba shares, some iSCSI mount points. I use a pair of intel gigabit NICs teamed together to allow samba to saturate the client computer's gigabit connection. Forcing the client gigabit cards to full duplex fixes a lot of the stupid speed problems associated with network transfers.

Currently the 1.5 TB drives are in a RAIDZ2 array with dedupe turned on, the 4 1TB drives are in a RAIDZ1 with dedupe on, and the rest are for virtualbox, the OS, and unassigned stuff. The SSD is used as a hybrid storage L2ARC drive, where the in-RAM cache will kick stuff to when it runs out of space, so I in effect have an 82GB read cache with sub millisecond access times. God I love ZFS.

Backup consists of me borrowing the IT department's Dell LTO-4 drive and making a monthly backup of all my stuff to tape, which gets thrown in a media rated fireproof safe at work. Backups currently take about 2 full days changing tapes every ~3 hours for 12-14 hours each day. Tapes are like $35 each now, so they're finally competitive with magnetic media for long term storage. I figure if I'm stuck making backups of all our stuff at work to that stupid tape drive, I should at least use it over the weekend to backup all my stuff too.
 
Last edited:
10.32TB (Retail)

Rosewill R901-P BK
OCZ StealthXStream OCZ700SXS 700W
GIGABYTE GA-MA785GM-US2H
AMD Phenom II X2 550 Black Edition (stock cooler)
OCZ Gold 4GB (2 x 2GB) DDR2 800 (OCZ2G8004GK)
Supermicro AOC-SAT2-MV8
1 x 320gb WD (OS Drive) (WD3200JD)
1 x 1TB Seagate Barracuda 7200.11 (ST31000333AS)
5 x 1TB WD Black (WD1001FALS)
2 x 2TB Hitachi (HDS722020ALA330)

Windows 7 Pro x64

This doubles as my HTPC (running Media Portal) and serving up media to the other systems in the house (just 3 laptops and my gaming desktop).

2uek13l.jpg

wtlzb.jpg
 
Amount of storage in the following system: 18.081TB advertised

Case: Norco 4220
PSU: Enermax ERV1050EWT Revolution85+ 1050W
Motherboard: Intel S3210SHLC
CPU: Q6600
RAM: 8GB DDR2 ECC
GPU (if discrete): Onboard
Controller Cards (if any): LSI SAS3442E-R plus HP SAS expander
Optical Drives: None
Hard Drives (include full model number): 6*HITACHI 0F10311 2.0TB, 6*HITACHI Deskstar 7K1000.B 0A38016 1TB, 1*80GB 2.5" boot disk, 1GB ACARD ANS-9010B.
Battery Backup Units (if any): Little 350VA APC for now, working on it.
Operating System: OpenSolaris b134

This machine serves home directories and media for me and my roommates over CIFS and NFS. Important stuff is backed up to CrashPlan, all 1.6TB of it. Things are mostly sorted into folders (TV, Music, and so forth), but home directories are cluttered however that user wants things :rolleyes: Really important things (right now, this means "subversion repository with schoolwork in it") are stored on my shared-hosting machine, which sends updates to this machine, and from there they're backed up to CrashPlan.

I also keep an up-to-date copy of the latest repository on half a dozen machines. In short, if there comes a point where I can't get a copy of that data, I probably have bigger problems to worry about, like Godzilla destroying Maryland. And California, and Utah.

Originally, everything was on the 6 1TB disks, which are arranged as a single raidz2 vdev; I got the 2TB disks in about a week ago and I'm in the process of verifying that they're not DOA and doing some stress tests. Then all the data will get moved off the 6*1 pool and onto the 6*2 pool, and then I'll destroy the 6*1 pool and add those disks as a second vdev to the new pool. This will give me 12TB of real capacity.

I have a solid-state RAM-based log device, the ACARD ANS-9010B on the 6*1 pool. It helped NFS performance under load a huge amount, but at $250, it better (I used 1GB of ECC memory that I had lying around, and that's been plenty so far). I also have a new 80GB Intel SSD that I might play around with as a L2ARC device. I'll turn on dedup and re-send the pool and see how things go. I think it's pretty promising; it did 11,000 4k random reads in my desktop box.

On to the pics. Sorry, they're all lousy, but hopefully they get the idea across. The ACARD SSD is mounted on the lid of the case:
router%20and%20acard.jpg

The case looks like every other 4220, but this one is mine:
case%20front.jpg

The router just barely clears the SAS expander, but it does:
router-repositioned.jpg


For completeness' sake, here's the router config:
I modded the PicoPSU to connect to a standard 4-pin Molex connector, and it works quite nicely. The router handles a full 100 megabits of Internet traffic after the upgrade to the Intel NIC; the onboard Realteks choked at around 30.

I don't even want to think about how much this whole setup cost to build, but it's compact, stable, fast, and extremely functional. There's room for 8 more disks, and plenty of power and cooling to go around.
 
I finally made it to 10 TB.

Case : Antec Three Hundred
Power Supply : Corsair HX520w 520W
Motherboard : Gigabyte GA-MA74GM-S2
Mem: Corsair TWIN2X4096-6400C5 DDR2 800 - 4GB (2 x 2GB)
CPU: AMD Athlon X2 4850e

whs_2010_04.jpg
 
"BEN-NAS"
20tb Advertised, 18.6tb Formatted

Finally completed my file server :D I really need to tidy the bottom part of the case haha ...

Click the pics to make them bigger!

CPU: Intel E5200
Cooler: Arctic Cooling Freezer 7 Pro Rev. 2
Mobo: ASUS P5Q WS
RAM: 2GB PC6400
Case: NZXT Whisper (build quality SUCKS but it's quiet...)
PSU: 800w Modular (FSP I think)
GFX: ATI 4350 PCI-E
SATA Controller: Raidcore BC4852 PCI-X (no RAID, just for connecting drives)
Drives: 8x WD 2TB Green 64mb WD20EARS, 4x Seagate LP 1.5TB, 500GB Seagate (UnRAID Cache Drive), Boots from 2GB JetFlash USB drive
OS: unRAID 4.5.3 Pro (Will update screenshot soon once array has finished building

2uiwyty.jpg


11aux6p.jpg
 
Last edited:
11.08TB

Coolermaster HAF 932
Corsair 850W
Intel DX58SO
Core i7 920
6GB Corsair DDR3
BFG GTX 260 896MB
4 Port Silicon Image SATA PCI Card
SATA CD/DVD-R/W Drive
6x 1.5TB Seagate ST31500341AS
1x 1TB Seagate ST31000340AS
1x 640GB Seagate ST3640323AS (OSX Boot drive)
1x 500GB Maxtor (Windows boot drive)
Mac OSX 10.6.2 / Windows 7 Enterprise x64

I use my computer as a media server at home. It streams to an xbox360 and Playstation 3. I also have several computers backing up to it. I only use Windows for gaming.

Pics:

DSC_0397-1.jpg

DSC_0403-1.jpg

DSC_0404-1.jpg

DSC_0395-1.jpg
 
Last edited:
So here I come :D
all system in the rack: 22,7TB


(click the image - if you want to see more..)
#2

(click the image - if you want to see more..)

My 12TB System ( 8x 1,5TB@RAID6 ) Running Windows Storage Server 2003 x64 R2

(click the image - if you want to see more..)

Inside is a Core 2 Xeon with 2x 3Ghz and 8GB DDR2 800Mhz ECC and an Adaptec 5805Z.
A single 750GB Disk is used of OS.
3x Intel NIC@3GbE

And here my second Storage System:

(click the image - if you want to see more..)

5x 1TB @Highpoint 3320 at RAID5 and 2x 200GB @RAID 1 for OS.
2x AMd Opteron 275 Dual core 7GB DDR1 ECC reg
3x Boardcom NIC@3GbE


Hope you enjoyed it ;)

For more pictures and information visit my flicker account or my blog...

MAFRI
 
Last edited:
Total Storage 24.75TB

NAS:
Amount of total storage: 20.25 TB
Amount of usable storage: 16.25TB
Case: Norco 4220
PSU: Corsair HX650
Motherboard: Tyan S5207 (Intel I3100 Based)
CPU: Intel Mobile Celeron 410
RAM: 1*2GB ECC DDR2-400 Reg
GPU: Integrated ATI ES1000
Controller Cards: Adaptec 52445
Optical Drive: ESATA External
Hard Drives:
Qty 10 - WD20EADS (TLER enabled) 2TB Raid 6
Qty 1 - WD2500BEVT 250GB
Battery Backup Units: APC Smart-UPS 700VA
Operating System: Centos 5.4 X86
Software: MediaTomb

Very happy wwith performance, I get 100-108Mbyte/sec to and from Windows clients (Providing the HDs on the client can keep up), the little Celeron barely breaks 50% on its single core. Streaming Reads and Writes are about 450Mbyte/sec.

Older Server:
Amount of total storage: 4.5 TB
Amount of usable storage: 3TB
Case: Supermicro 733T
PSU: Supermicro 450W
Motherboard: Tyan S2507 (Serverworks LE-T based)
CPU: Single Intel P3-S 1.13Ghz
RAM: 3.5GB PC133 ECC Reg
GPU: Integrated ATI
Controller Cards: Areca 1120 PCI-X
Optical Drive: NEC ND-4571A DVDRW
Hard Drives:
Qty 3 - Seagate ST31500341AS 1.5TB Raid 5
Qty 1 - WD360GD 36GB
Operating System: Windows 2000 Server
 
Last edited:
Total Storage 13.76TB

R1 = RAID1/mirror
R5 = RAID5

Workstation:
Mac Pro 2009 "Nehalem"
- CPU: Xeon 2.66GHz Quad (8 Threads)
- RAM: 12GB ECC Regged RAM (3x 4GB Samsung)
- GPU: 2x GT120 Graphics Card
- PSU: 1.000 Watt PSU
- Controller: Sil3132 eSATA PCIe x1
- ODD: DVD+-R/W
- Displays: 1x 30" Apple Cinema HD Display + 2x Samsung 226BW 22" Displays

--------
80 SSD / System / Intel Postville X25-m G2
1500 R1a / Samsung
1500 R1a / Samsung
640 R1b / WD Caviar Blue
640 R1b / WD Caviar Blue
--------
4280 Advertised
2200 Available

STORAGE eSATA / Onnto DataTale external RAID5 Box
--------
1000 R5 / WD Caviar Green
1000 R5 / WD Caviar Green
1000 R5 / WD Caviar Green
1000 R5 / WD Caviar Green
--------
4000 Advertised
2700 Available

BACKUP eSATA / Onnto DataTale external RAID5 Box
--------
1000 R5 / WD Caviar Green
1000 R5 / WD Caviar Green
1000 R5 / Hitachi
1000 R5 / Hitachi
--------
4000 Advertised
2700 Available

Ext A FW800 / MacPower external FW400/800, USB2, eSATA Case
--------
640 / WD Caviar Blue
--------
640 Advertised
640 Available

Ext B FW800 / MacPower external FW400/800, USB2, eSATA Case
--------
640 / Hitachi
--------
640 Advertised
640 Available

Storage @Mac Pro
--------
13560 Advertised
8880 Available


Server (Debian Lenny / DEV Machine)
- CPU: AMD Athlon X2 240e
- RAM: 6GB ECC RAM (2x 1GB Nanya + 2x 2GB Kingston)
- GPU: onboard
- PSU: 450 Watt Tagan PSU
- NIC: Intel Desktop 1000/Pro CT PCIe x1
--------
200 R1 / Samsung
200 R1 / Seagate
--------
200 Advertised
200 Available

PS. Photos coming soon, or check here on flickr: http://www.flickr.com/photos/37773250@N02/sets/72157623537965887/
 
Last edited:
Status
Not open for further replies.
Back
Top