The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.

nry

Limp Gawd
Joined
Jul 10, 2008
Messages
409
That Sun SAN looks awesome! 22 drives in raidz2 on the other hand, not so awesome haha

Just waiting on my Areca 1882ix-24 and should have a nice post for in here. Should have 56TB advertised 42TB usable

I have 16 Hitachi 3TB disks, and 2 have failed on me, one of the failed ones Im not too sure if it was a false report though still haven't investigated. Where as the other simply died and was replaced under warranty
 

patrickdk

Gawd
Joined
Jan 3, 2012
Messages
744
Yes, 22disks in a single vdev as long as used for streaming is ok, still going be painful some with that small blocksize.

Used for anything else will be dead slow.

Thankfully I guess your complete working set fits in ram, when it slightly goes over your installed ram, you will feel like you jumped off a cliff, everything will be dog slow, if it moves at all.
 

unhappy_mage

[H]ard|DCer of the Month - October 2005
Joined
Jun 29, 2004
Messages
11,455
how are you finding/liking the hitachi's? any comments on reliability? :)
Just fine. No disk failures at all. I ordered the 1TB disks in Feb 2009, the 2TB disks in March 2010, and the 3TB disks in August 2011. I wish they were still in business. I think I'm gonna go for Toshiba next time around, but it won't be the same.
 

Matthew Kane

Supreme [H]ardness
Joined
Dec 1, 2007
Messages
4,233
Eh, not sure how reliable toshi desktop drives are but there 2.5inch hard drives are completely unreliable prone to fucking up from normal use within the first few months or bit over the year.
 

Welsper

Gawd
Joined
Jun 30, 2005
Messages
737
Just fine. No disk failures at all. I ordered the 1TB disks in Feb 2009, the 2TB disks in March 2010, and the 3TB disks in August 2011. I wish they were still in business. I think I'm gonna go for Toshiba next time around, but it won't be the same.
Eh, not sure how reliable toshi desktop drives are but there 2.5inch hard drives are completely unreliable prone to fucking up from normal use within the first few months or bit over the year.
AFAIK, the Toshiba 3.5" drives are (basically) rebranded Hitachi drives, aren't they?
 

nry

Limp Gawd
Joined
Jul 10, 2008
Messages
409
AFAIK, the Toshiba 3.5" drives are (basically) rebranded Hitachi drives, aren't they?

Don't WD own HGST who own Toshiba?

Did read somewhere on here about them being rebranded drives though. Not seen an official source for that
 

Welsper

Gawd
Joined
Jun 30, 2005
Messages
737
Don't WD own HGST who own Toshiba?

Did read somewhere on here about them being rebranded drives though. Not seen an official source for that

IIRC WD bought HGST but was required to sell their storage stuff to Toshiba, or something like that. I can't remember anymore :(
 

tycoonbob

Limp Gawd
Joined
Jul 29, 2012
Messages
157
IIRC WD bought HGST but was required to sell their storage stuff to Toshiba, or something like that. I can't remember anymore :(

Pretty much. WDC bought HGST, but to satisfy the FTC, sold the desktop line (aka, Deskstar) to Toshiba (to avoid a duo-opoly with Seagate, who recently bought Samsung). Toshiba now own all Desktop HGST assets, and the Deskstar drives are still made in the same facility, but with a new label. They are the same hardware (as far as I know), but may have a different firmware.
 

darkdub

Weaksauce
Joined
Mar 22, 2011
Messages
102
Current file server / VM host setup:

Case: CoolerMaster Centurion 590
Mobo: Supermicro socket 1155
Proc: Xeon E5-1650
Memory: Crucial DDR3-1600 56GB (one bad stick, replacement on hand but haven't powered down yet)
Enclosures: Supermicro 5-in-3 hot-swap tray (CSE-M35T-1B)
OS: Linux (CentOS 6.3 w/ custom kernel incl. Xen and bcache)
Drives / Arrays
Boot Drive: Intel 30GB SSD

Cache Drive: OCZ Vertex 3 64GB SSD

Array 1 (RAID-6)
7 x WD Green 2TB
2 x Seagate "green" 2TB
1 x Seagate 7200 RPM 2TB (pinch replacement)

Array 2 (RAID-6)
6 x WD Green 3TB

Total space: 38 TB
Usable space: 14.54 TiB + 10.9 TiB = 25.44 TiB (or 25.5TiB if you count the SSD's)


Pics (please excuse the mess....)

PXnc7Wsl.jpg

Case front, all visible drives spinning away...

Sv4Bdphl.jpg

Inside... how do you like the drive velcro'd to the PS?

vNp1N2yl.jpg

Or the two SSD's screwed into the fan grille's at the top?

9z8qW7el.jpg

All at a whopping 250ish Watts when all spun up.

YdM4uTN.png

Screenshot of /proc/mdstat


Overall I'm very happy with my setup. It's more than fast enough to saturate the two gigabit links and serve the various TVs, tablets and music-playing devices throughout the house. Thanks to the SSD cache (through bcache) in write-back mode, my virtual machines are extremely snappy despite the green backing RAID.

I will say though that, given my experience with failing WD green drives, I will be much less likely to buy the bottom-barrel model in the future. My WD Black / RE drives have all been fine or, at least, within understandable tolerances. Of the green drives, I started out with 10 WD Green 2TB drives, of which, 7 have been sent in for RMA. Of the drives returned (over time), 3 were DOA and had to be RMA'd again.

Of the 3TB drives, every single one was bad on delivery, either showing many bad sectors or extremely slow performance in both Linux and Windows machines. Additionally, of the replacements, 5 of the 6 were either DOA or failed within the first 7 days.

Much of this occurred around the time when factories were closing / having difficulties due to the flooding. The replacements seem to be OK, but as they die (hence the 3 non-WD 2TB drives), I'm replacing them with whatever other brand is available at the local micro center at the time.
 
Last edited:

Goros

n00b
Joined
Feb 14, 2013
Messages
52
Total storage: 25.2TB

Case: Chenbro SR-10769 w/10-3.5" hotswap bays, 2-2.5" internal bays
Mobo: Asus P9X79-WS
CPU: i7-3930k @ 4.1GHz 1.31v w/Noctua NH-D14
RAM: 32GB Patriot Black Mamba 1600 9-9-9-24 1t 1.5v
PSU: Silverstone ST1500 Modular
GPU: GTX650 1GB DDR5
HBA: IBM M1015 Flashed to LSI 9211-8i in IT Mode
SSD1: Vertex4 128GB (OS)
SSD2: Vertex4 128GB (Apps)
HDDs: 5 @ 2TB Hitachi GST Deskstar 7K3000 7200RPM; 5 @ 3 TB Seagate 7200.14 7200RPM
Optical: Slim Optiarc DVD-RW; External Optiarc DVD-RW (USB3.0); External LG BDXL-RE (USB3.0)
TV Tuner: Hauppauge HVR-2250 Dual Tuner w/USB Hauppauge 23064 PCTV IR Blaster/Receiver
OS: Win7 x64 Ultimate SP1
Background apps: FlexRAID, Syncback Pro, Diskeeper 12 Pro, Hard Disk Sentinel Pro

Connected via HDMI to my Denon AVR 4311ci, configured to pass 0-255 and bit exact audio via bitstream.

Case w/USB3.0 SDXC multi card reader
161597

Hotswap bays x10 - two 4 bay Sata 6.0Gbps hotswap backplanes & 2 Sata 6.0Gbps hotswap 5.25" bay converters
161598

Silverstone ST1500 Modular PSU, CPU, Mobo
161599

Hotswap backplanes powered, RAM installed, SAS HBA installed.
161600

Noctua NH-D14 fit perfectly
2013-03-05_15-24-04_381_zps6894d7e7.jpg

GTX650 & M1015 HBA
2013-03-05_15-24-10_514_zps2f66f2d9.jpg

Running...
2013-03-06_16-58-15_715_zpse7a554a2.jpg

Fully populated and with media on it!!
2013-03-28_22-26-50_136_zpsb2943d14.jpg


So, for the actual frontend appearance...this is what WMC7 with MediaBrowser and some plugins looks like...
2013-03-29_13-18-40_765_zps54b4324c.jpg

Movies appearance
2013-03-29_12-39-22_79_zpsb3413efc.jpg


For my storage array, I use FlexRAID drive pooling with parity. It takes my 10 HDD's and combines them into one large pool with 2 parity drives (I can add more later, a-la RAID60). You must use your largest HDD for parity, so in an array with 2TB and 3TB drives, you must dedicate your 3TB drive (or both 3TB drives in my case). I used GPT tables for all 10 drives (with MBR on the SSD's) with 64k clusters (4k on the SSD's)

Here is a snapshot of the servers drive list (note that both my external optical drives are off at this point)
2013-03-29_12-50-47_859_zpsd9e00d70.jpg

What I've ripped so far doesn't take much space
2013-03-29_12-51-18_790_zps7e1dc84f.jpg

And the view in Flexraid of my Storage and Parity drives
2013-03-29_12-52-46_978_zps95a5d7a6.jpg

Hard Disk Sentinel runs in the background and monitors the SMART status of all drives in real time and reports issues immediately so data risk is minimal.
2013-03-29_12-58-51_895_zps428bf93a.jpg
 
Last edited:

Goros

n00b
Joined
Feb 14, 2013
Messages
52
I transcode plex to 3-4 devices as well as stream to an Xbox extender, plus I run the system as an HTPC connected to my Denon 4311ci via HDMI and use MPC-HC, madVR, ReClock, and SVP.

I also have 3 optical drives that I use to rip & encode via handbrake and MakeMKV, which I open multiple instances of. Since this is my first HTPC or server of any kind, I have a lot of DVD's and Bluray's to rip.

So yes. This should last me for ~5-6 years, hardware failures notwithstanding.

...and, the title is "showoff thread".
 

EnderW

[H]F Junkie
Joined
Sep 25, 2003
Messages
11,095
list has been updated, but the links are messed up, I will get them sorted out asap
 

Argentum

Weaksauce
Joined
Nov 14, 2006
Messages
89
Amount of total storage: 90TB

Case 1: 50TB
Case: Antec 1200 + 3 x Supermicro CSE-M35T-1B 5-in-3's + 1 x Icy Dock 5-in-3 (which I'll replace whenever the fan breaks. It's leftover from my unRAID days)
PSU: Corsair AX 850
Motherboard: SuperMicro X9SCM-F-O
CPU: Intel Xeon E3-1240
RAM: 6 x 2GB fb-dimms
Controller Cards: SuperMicro AOC-USAS2-L8i
Controller Cards: LSI SAS9201-16e
Controller Cards: LSI SAS9201-16i
Controller Cards: Qlogic QLE2462 FC card for iSCSI over FC to other servers
Boot Drive: 2 x Seagate ST95005620AS 500GB
Hard Drives: 4 x Western Digital WD30EARX 3TB
Hard Drives: 3 x Western Digital WD30EFRX 3TB
Hard Drives: 3 x Hitachi HDS5C3030ALA630 3TB
Hard Drives: 10 x Hitachi HDS5C3020ALA632 2TB
Battery Backup Units: CyberPower CP1350PFCLCD PFC Sinewave UPS
Operating System: Solaris 11.1

Case 2 via SAS Expander: 40TB
Case: Antec 1200 + 4 x Supermicro CSE-M35T-1B 5-in-3's
PSU: SeaSonic X750 Gold
Controller Cards: Chenbro CK13601
Hard Drives: 4 x Hitachi HDS5C3020ALA632 2TB
Hard Drives: 1 x Hitachi HD32000 IDK/7K 2TB
Hard Drives: 2 x Seagate ST2000DL003 2TB
Hard Drives: 4 x Seagate ST32000542AS 2TB
Hard Drives: 3 x Western Digital WD20EARX 2TB
Hard Drives: 6 x Western Digital WD20EARS 2TB

I replaced all the Supermicro fans with 92mm Noctua's to cut down on noise. Runs around 315W idle.

A straightforward design over two similar chassis actually. I've had this server over a year, reaching 90TB by upgrading a 20TB pool to 30TB this past November. I use the system primarily for media storage but also to experiment with presenting LUNs using iSCSI over FC to Hyper-V servers. This is for home use ... but also self-education in the IT field for enterprise class applications.

Chassis 1:
WP_20130329_007_zpsd4be09e1.jpg

(hmm... need to clean some dust out there)

Chassis 2:
WP_20130329_005_zpsf719d6aa.jpg


Inside Chassis 1:
WP_20130329_004_zpsaa610bd4.jpg


Mishmash of mostly green drives (can you spot the one 7200 RPM in there :p )
disk1_zps29cdf527.png

disk2_zps43a05b16.png



(I actually have a Intel Micro-ATX board in the second chassis running a Hyper-V server, because I figure it's a waste of space otherwise ;) but it's not needed for the storage server. I do have to be careful rebooting it because its power state affects the power to the drives in the second chassis.)

I bought most of my 2TB Hitachi's around $60-$70 back in the heyday before the Thailand floods.

I partitioned into four unique 10-drive pools using RAID-Z2:
poolsizes_zps1ae972fa.png


The Antec case's PCI brackets actually have gaps in them that I stuck screws through into spare 3.5" SSD trays used to hold the 2.5" HDD boot drives for the Solaris system.

There is some sentiment here at [H] that SAS expanders should be avoided with ZFS. When I was on the original Solaris 11, I did have problems when I added the second chassis- upon reboots I'd have pool corruption in the pools on the SAS expander and would need to run "zpool clear" over and over until the pool's errors went away. I originally had an HP SAS Expander which I replaced with the Chenbro in desperation to no avail.

But since upgrading to Solaris 11.1 I have had no issues whatsoever... which implies to me it was a software/driver bug. I don't pay for support so I'm at Oracle's mercy for releases.

So now I see the mishmash of drives as a testament to ZFS's ability to use and maintain consumer SATA drives even off of a SAS expander. I'm not into spending hours playing with ashifts and such to optimize performance if it's "good enough" and so my internal speeds of pools are not consistent: the best performing pool just based on scrub speed is the one with all Hitachi drives in the first case; the second best the 10x 3TB pool in the first case; then it drops down for the pools in the second case. My scrub speeds are around:
Most uniform pool (andromeda: 10x2TB Hitachis): 14.7T allocated scrubbed in 7h 28m
Mix of 3TB drives on local controller: 19.5T scrubbed in 12h
Mix of oldest 2TB drives on SAS expander in second case: 15.9T scrubbed in 12h 40m
Mix of 2TB drives on SAS expander in second case: 13.9T scrubbed in 8h 52m

The design is theoretically scalable to three more cases without daisy chaining because I used a LSI SAS9201-16e, but the heat and power probably keep me from going any further. I came up with this design back in early to mid 2011 because I figured by now 4TB drives would be $60 and 8TB drives would be the $200 top of the line. I figured I could just replace drives out to grow the pools (e.g. replace 10 2TBs with 10 4TBs for, say, $600 and use the old ones as offline storage.) C'est la vie. :rolleyes:
 
Last edited:
D

Deleted member 82943

Guest
I transcode plex to 3-4 devices as well as stream to an Xbox extender, plus I run the system as an HTPC connected to my Denon 4311ci via HDMI and use MPC-HC, madVR, ReClock, and SVP.

I also have 3 optical drives that I use to rip & encode via handbrake and MakeMKV, which I open multiple instances of. Since this is my first HTPC or server of any kind, I have a lot of DVD's and Bluray's to rip.

So yes. This should last me for ~5-6 years, hardware failures notwithstanding.

...and, the title is "showoff thread".

makes sense
 

RangerXML

Supreme [H]ardness
Joined
Apr 16, 2006
Messages
6,394
Currently on my main system:
1x 1TB WD Black (OS, Windows 8 Pro x64)
2x 2TB WD Green (2TB)
2x 1.5TB Samsung (3TB)
1x 4TB WD MyBook
Storage_zps9cd00783.jpg


Planned upgrade for the coming month:
1x 4TB WD Black (OS)
4x 2TB WD Green (Raid 5, 6.4TB External USB3.0/eSATA box)
1x 4TB WD MyBook

Gonna keep the 2x 1.5TB drivers with the current system as back up and storage off site.
 
Last edited:
D

Deleted member 82943

Guest
Currently on my main system:
1x 1TB WD Black (OS, Windows 8 Pro x64)
2x 2TB WD Green (2TB)
2x 1.5TB Samsung (3TB)
1x 4TB WD MyBook
Storage_zps9cd00783.jpg


Planned upgrade for the coming month:
1x 4TB WD Black (OS)
4x 2TB WD Green (Raid 5, 6.4TB External USB3.0/eSATA box)
1x 4TB WD MyBook

Gonna keep the 2x 1.5TB drivers with the current system as back up and storage off site.

A 4tb boot drive why not smaller ones say 2 in raid 1?
 

RangerXML

Supreme [H]ardness
Joined
Apr 16, 2006
Messages
6,394
Its going to be be in a little Shuttle case so I only wanna have 1x HDD in it and I was reading the 4TB Blacks actually perform better then the 2TB, plus it will be hopefully big enough that I can just forget about it as you can see with my 1TB OS drive. I was considering a 256GB SSD and then use the 4TB driver for all my Steam games and User directories, but that would add about a $300 premium on top of my existing cost. I understand about the inefficiency, or the possibility there of, I had never even considered it, until a friend had me do a 4TB boot drive on his PC (I had suggested the 2TB and he got a 4TB) and its working surprisingly well. I just hope to god it doesn't need to run a diskchk.
 

balance101

Limp Gawd
Joined
Jan 31, 2008
Messages
411
do you guys get deals on Supermicro 5-in-3 hot-swap tray (CSE-M35T-1B) ?

because they cost 100 each, and four of them is 400, pretty close to a norco 24 bay case but ofcourse supermicro quality is better

if theres a place that sells cheaper point me to it :D
 

Blue Fox

[H]F Junkie
Joined
Jun 9, 2004
Messages
11,814
Just watch eBay. There was a listing for 5 of them that finished 3 days ago for $65 a piece.
 

Argentum

Weaksauce
Joined
Nov 14, 2006
Messages
89
do you guys get deals on Supermicro 5-in-3 hot-swap tray (CSE-M35T-1B) ?

because they cost 100 each, and four of them is 400, pretty close to a norco 24 bay case but ofcourse supermicro quality is better

if theres a place that sells cheaper point me to it :D

Not as good as eBay from the prior post, but I usually pay around $90-$95 using coupons and such @various online vendors (e.g. thenerds.net, rakuten, or SuperBiiz).

I know I could get a 24x Norco for less than my design (Antec 1200 + 4 CSE-M35T-1B + 4 x Noctua 92mm fans) but my system is hellalot quieter. If I get a house with a basement and proper cooling someday where I can rackmount maybe I'd rethink.
 

SedoSan

Limp Gawd
Joined
Aug 29, 2012
Messages
145
I've started a RAID build around a year ago after doing lots of research on how to get started. I found a very cheap used LSI 9261-8i with BBU on ebay in good condition and relatively new so I grabbed it.
I have started with 4 3TB WD RED HDDs and to my rush I didn't get a dedicated server case, instead I've installed them on my gaming rig.
I got a kingwin 3x5.25" bay to 4 HDD hotswap and put them in. Looked nice

422592_10152573634750603_1624742585_n.jpg


I also did some tests for speeds and these are the results:

487906_10152519795760603_579080865_n.jpg


Setting the array to RAID5 limited my storage to 9TB but I filled them so fast I found myself in a need of expansion. So I got 3 more HDDs (the 7th one is now added and the array is being reconstructed as I type), bringing me to a total of 18TB.

Running 2x dual GPU cards really made my case VERY crowded! That's why i'm planning to get a dedicated server case. And here is a look of my desktop:

24266_10152573634940603_93257349_n.jpg


**** Edited ****
I had some questions that I put here, but decided it would be better to make a new thread for them.
*****************
 
Last edited:

gmutale

Limp Gawd
Joined
Jan 22, 2011
Messages
211
@SedoSan - how on earth do you sit so close with those big monitors, esp. so much to your peripheral?! :/
 

SedoSan

Limp Gawd
Joined
Aug 29, 2012
Messages
145
@SedoSan - how on earth do you sit so close with those big monitors, esp. so much to your peripheral?! :/

I dunno.. it feels normal~ >.>; If i wanna work on either monitor I just turn my head, but my main work is in the middle screen. Though I use this setup mainly for Eyefinity games. The game experience is just so different! :D
 

Welsper

Gawd
Joined
Jun 30, 2005
Messages
737
Nine Vertex 4s. One 256GB for OS/Boot on the Intel SATA3 6gbps controller and eight 512GB in RAID-0 on an LSI controller integrated into my ASRock X79 Extreme11. I also have a 4TB Hitachi 7200RPM drive for backups.
Holy fcuk... :eek:
 

gmutale

Limp Gawd
Joined
Jan 22, 2011
Messages
211
Nine Vertex 4s. One 256GB for OS/Boot on the Intel SATA3 6gbps controller and eight 512GB in RAID-0 on an LSI controller integrated into my ASRock X79 Extreme11. I also have a 4TB Hitachi 7200RPM drive for backups.

this thread is for people with 10 or more TERAbytes of storage.. your array, impressive at speed as it is, falls well short of the criteria *thumbdown*
 

Neb

2[H]4U
Joined
Mar 13, 2000
Messages
3,320
this thread is for people with 10 or more TERAbytes of storage.. your array, impressive at speed as it is, falls well short of the criteria *thumbdown*

Agreed. It's mighty fast but 10TB+ it isn't :p
 

AdamK47

Limp Gawd
Joined
Feb 7, 2004
Messages
346
this thread is for people with 10 or more TERAbytes of storage.. your array, impressive at speed as it is, falls well short of the criteria *thumbdown*

Thank you. I edited my post to fit your requirements. *thumbsup*
 

Melakion

n00b
Joined
Nov 16, 2010
Messages
10
96TB + 48TB + 22.5TB +.08TB = 166.6TB

Dell Poweredge T110
650W Seasonic
Dell Poweredge T110 MB
Xeon X3450 2.67Ghz Quad Core
8GB Ram
Geforce GT8800, but it doesn't really need to be in there anymore.
Areca ARC-1880i
DVD-RW
Intel X25-M 80GB SSD
APC Smart-UPS 3000XL + Battery Pack
Windows 7 Pro

2x 1m SFF-8087 to SFF-8088 SAS Cables

Enclosure 1:
Norco RPC-4224
120MM Fan Wall Bracket
PC Power & Cooling Turbo-cool 1kw
PE-2SD1-R10-1 PICMG 1U Backplane
HP SAS Expander
15x Seagate 7200.11 1.5TB HD
22.5TB Drives, 15TB 12 Drive Raid 6 + 3 Hot Spare. 13.6TB Usable

Enclosure 2:
Norco RPC-4224
120MM Fan Wall Bracket
PC Power & Cooling Turbo-Cool 510 SLI
Chenbro AC UEK13601 SAS Expander Kit
24x Samsung HD204UI 2TB HD
48TB Drives, 44TB 24 Drive Raid 6. 40.0TB Usable

1m SFF-8088 to SFF-8088 Cable

Enclosure 3:
Norco DS-24E
Seasonic 650W
Areca ARC-8026 SAS Expander
5x Seagate ST4000DX000 4TB HD
19x Seagate ST4000DM000 4TB HD
96TB Drives, 88TB 24 Drive Raid 6. 80TB Usable

xPilHIC.jpg


Don't have a proper rack yet, eventually want to move it all into the basement but I need a 1/2 or 3/4 rack to fit down there. Haven't found a cheap one on craigslist yet.
The new Enclosure 3 is hanging off the Chenbro expander for now because it had a free SFF-8088 port. The Chenbro is only a 3G expander so I'd like to have it on one port and the Areca and HP Expander on the other since they are both 6G, but I need to put it all in a proper rack for that, or at least stack it all on the UPS's on the floor..

umvh4M3.jpg
eyLBUOz.jpg

zMgWfky.jpg

tXLdc6Q.jpg

C3Jr5ew.jpg

6wzV4XD.jpg

PoZIsum.jpg

9jtMrMe.jpg
 
Last edited:

nry

Limp Gawd
Joined
Jul 10, 2008
Messages
409
Oh my!

Love how your 'LitteRaid' is the size of my one and only raid haha
 
Status
Not open for further replies.
Top