The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
Not as kool as the majority of these setups in here but slowly growing & realizing that once you start you can't stop (expanding your storage that is).

CaseLabsMagnumMH10FileServer-Build2-2Resizedx1300.jpg


The system on the left is my File Server. This stays at home all the time & is now using my old setup I had a few years ago.

Current File Server Specs..

Intel i7 2700k (@ 4Ghz)
ASUS Maximus V Extreme
Corsair 16GB (4x4) DDR3 1600 (8-8-8-24 1N)
45.12TB - 15x 3TB's (WD Red's & Toshibas) & 120GB Corsair Force 3 SSD
LSI 9261-8i (+BBU) & Intel 24-Port Expander. (2x RAID5 Array's)
Intel PRO/1000 PT Dual Port NIC + Connected to Netgear GS724Tv3
Corsair AX1200
Magnum MH10

Current Main Rig Specs..

Intel 3930k (@ 4Ghz - Loop)
ASUS Rampage IV Extreme
G.Skill 32GB (4x8GB) DDR3 2400 Trident-X C10
2-Way SLI EVGA GTX 580 3GB
ASUS Essence STX
60.75TB - 18x 3TB's / 1x 4TB / 1x 2TB / 3x 250GB SSD's (Toshiba/Seagate/Samsung/Intel) - Only 28TB Shared/Hashed on Network
IBM M1115 & Intel 24-Port Expander (Single DIsk/Software Pooled)
Corsair AX1200
Magnum SMH10
Dell U3011

Spare/Backup Drive Storage Tub..

83.**TB - 22x 3TB's / 4x 2TB's / 6x 1TB's / 8x Less then 1TB's (Approx 3TB all up). All dated from 2009 to now. WD/Hitachi/Toshiba/Seagate/Samsung

Eventually I would like to get 2x Norco 4224's. I want to make sure that I don't lose any data even if I have a spare drive for my array. I've lost a chunk before from 'trusting' a RAID controller before, won't let it happen again. I also will have to invest in a 10Gbe NIC pretty soon for both systems since I just had to transfer approx 20-22TB over a 1Gbe NIC and that took nearly 2 days:(

When I first got this setup in the main rig I tried using the LSI 9261-8i controller on it and it absolutely hated it. The firmware would be non-existent & it basically wiped the storage configuration I already had set on it. I've also experienced many other issues with different cards on this board. I absolutely hate this Rampage IV Extreme but I've found the sweet spot of no issues & I plan on leaving it alone until ASUS release a newer version of this board. The Maximus V Extreme on the other hand is flawless, perfect & has always resolved any issues that the Rampage would have came up with...
 
SheHateMe said:
How to do you guys get hard drives in bulk like that?
Click to expand...
Amazon.com is your best bet. I know ZipZoomFly used to package drives like that, and I think NewEgg still does if you order large numbers of them (8+). CDW and other "professional" distributors will almost always package bulk orders that way. The last time I purchased bulk drives, I bought retail at Best Buy. Cheaper than online, and easier to pick up/return.
 
Hello,

Small update of my small setup (compared to yours)



Lian Li PCQ25 itx
PSU 350W Gold
P8H67-i
i5 2500
8GB DDR3
1*256GB SSD + 7*4TB (WD RED ERFX, HITACHI 5K4000, SEAGATE ST4000)
OS : Windows Server 2008R2 Sp1

Few benchmarks of HDDs used

HITACHI 5K4000

SEAGATE ST 4000

WD RED 4TB
 
Last edited:
greeneye said:
Not as kool as the majority of these setups in here but slowly growing & realizing that once you start you can't stop (expanding your storage that is).
Click to expand...

Very nice setup! So clean. I see you have a dual port NIC. Could you setup some kind of link aggregation and get 2Gbps on LAN? 10gb equipment still ain't cheap.
 
A few of the local stores around me will sell drives in bulk if you give a down payment they will order up to 50 at a time for you.

I will not ever buy a hard drive from best buy unless 1 its in a caged area or 2 I make them grab it from the back of the store.

I know some of you may think Why be a jerk about this?
Well around me it seems that a lot of kids pick up a 3tb external drive and throw it on the ground when their parent says no or they will throw it into a basket like its a pillow or a toy and it just makes me cringe. I am not saying best buy is at fault here for this.
 
Just wanted to share my new build for ESXI home lab.
vm's im running at the moment

Windows Server 2012 domain controller
OmniOS napp-it 12TB raw
unRAID 5.0 5TB raw
OpenMediaVault running plex, sabnzbd, sickbeard, couchpotato, transmission
Newnab Server Ubuntu 12.10 server
Windows 8
Pfsense
Vcenter


Case - NZXT Source 210
PSU - Corsair HX750 750W
Motherboard - Supermicro X9SRL-F-O
CPU - Intel Xeon E5-2658 2.1 GHz C2 ES
RAM - Kingston ValueRAM KVR16R11D4/16HA DDR3-1600
Controller Cards - IBM M1015 flash to LSI SAS 9211-8i
Network Card - Dell Intel PRO/1000 VT Quad Port
Hard Drives - 4 x Western Digital Red NAS WD30EFRX 3TB
5 x Seagate/Hitachi 1TB
SSD Hard Drives - SAMSUNG 840 Pro 128GB
OCZ Vertex 4 256GB
OS - ESXI 5.5

02mlmRB.png


8vFE7BK.jpg
 
Updated my Filestorage/ESXI host a couple of months ago just getting to finish it over the coming weekend.

ESXI host 1

i7 4770
GigaByte Z87X-UD3H
32GB Ram
X-Case 4220
LSI SAS3201-16i
3 NIC's
120gb SSD corsair force
250gb WD
500GB WD
2TB Samsung (backup for VM's)
sWjz1fnH



VM's

Win7 x 2 = odd jobs mostly testing out applications, game servers etc

Win 8 = Testing it out not convinced with 8 yet.

Ubuntu = used for home automation

Couple extra server 2012 boxes loaded ready for setting up AD etc.

Windows 2012 Server- used for File Server, SQL database (XBMC), Sabnzb etc..

These are set into spanned drives (its only for media storage and i have 80% backed up off site at a friends so im not overly worried. with the price of drives still high its not economical to buy hard drives monthly to keep for backups!)
3X Seagate 4TB drives
2x Seagate 3TB
5X Samsung 2TB
2X Hitachi 2TB
1X Hitachi 1TB
Total storage - 31TB

ESXI Host 2 - Not yet in service will be used to play with Gfx Passthrough and centralizing my XBMC clients.

Xeon E3-1225
8GB Ram
Tyan S5515
Yet to buy an SSD for this


HP Microserver - Had this as a backup box now redundant, not sure if i should sell this or not, as they are great little servers.
NL36
8GB
WHS2011
1x 2TB Hitachi
2x 1TB (cant remember what brand)
 
SheHateMe said:
How to do you guys get hard drives in bulk like that?
Click to expand...

Might seem odd, but unless I buy from a physical shop in the high street, my hard disks have always come packed like that, for the last 8+ years I've been buying them. Is that not typically the case in the US?
 
houkouonchi said:
And some pics:
Click to expand...

Well this really sucks.. Someone broke into my house. They stole two laptops (one was my sisters I was fixing), a computer i used as a HTPC box, and both of my TV's (42 inch and 50 inch 4k one) as well as my semi-auto shutgun (with one box of ammo) as well as my nexus tablet.

Thank *GOD* they didn't take any of my servers in my rack. I suspect I lucked out on that because they were all screwed in and it would have taken too long to take it all out but they did unplug all the power from the power strips and turned off one of the UPS's (probably because it was making noise). It really sucks but I count myself lucky.

Moving all my stuff to my dad's house until I can get a good alarm system as I thought with how much money worth of equipment is there its possible they could come back.
 
You are so lucky. They did not take all of them.
Few Days ago I lost one desktop pc from my office.
 
houkouonchi said:
Well this really sucks.. Someone broke into my house. They stole two laptops (one was my sisters I was fixing), a computer i used as a HTPC box, and both of my TV's (42 inch and 50 inch 4k one) as well as my semi-auto shutgun (with one box of ammo) as well as my nexus tablet.

Thank *GOD* they didn't take any of my servers in my rack. I suspect I lucked out on that because they were all screwed in and it would have taken too long to take it all out but they did unplug all the power from the power strips and turned off one of the UPS's (probably because it was making noise). It really sucks but I count myself lucky.

Moving all my stuff to my dad's house until I can get a good alarm system as I thought with how much money worth of equipment is there its possible they could come back.
Click to expand...

That seriously sucks so bad! :( hope u can get all or even few items back at some point..
 
SheHateMe said:
How to do you guys get hard drives in bulk like that?
Click to expand...

I get my drives from Newegg. I suspect they get them prepackaged.

I have had bare drives:

wrapped in bubble wrap,

in clamshell cases inside of brown WD boxes,

in foam (as shown by the post on the last page),

in double clamshell cases,

in some sort of custom bubble wrap that had a slot in the side the drive fit into.

I suspect most of the packaging methods are approved for this usage and are chosen based on cost and availability.
 
houkouonchi said:
Well this really sucks.. Someone broke into my house. They stole two laptops (one was my sisters I was fixing), a computer i used as a HTPC box, and both of my TV's (42 inch and 50 inch 4k one) as well as my semi-auto shutgun (with one box of ammo) as well as my nexus tablet.

Thank *GOD* they didn't take any of my servers in my rack. I suspect I lucked out on that because they were all screwed in and it would have taken too long to take it all out but they did unplug all the power from the power strips and turned off one of the UPS's (probably because it was making noise). It really sucks but I count myself lucky.

Moving all my stuff to my dad's house until I can get a good alarm system as I thought with how much money worth of equipment is there its possible they could come back.
Click to expand...

:( Sorry dude, that is truly a sad situation. While you are lucky you shouldn't have to be. I was raised to simply not touch things that aren't mine without asking. I really really hope you're able to get some or all of your stuff back, I'm pulling for you.
 
I posted my old setup in here when I did that a few years back, however have done some shuffling...


Amount of total storage (if posting multiple systems)
Amount of storage in the following system 57.3tb

Case: NZXT Source 210 ELITE Midtower, Omnistor SE3016 x2
PSU: Corsair ax760
Motherboard Intel S2400GP
CPU 2x E5-2440
RAM 24gb g.skill 1600
GPU (if discrete)
Controller Cards (if any) IBM M1115 crossflashed
Optical Drives no
Hard Drives (include full model number) See below
Battery Backup Units (if any) none yet
Operating System openindiana
(A short paragraph here describing what you use your storage for and how you handle backups and organizing would be nice.) Still in infant states. Storage was used to decom old storage arrays/whs box. Iscsi target setup for virtual machine data. Much is still in development/implementation.

Code: 
  pool: iscsi.1tb
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM     CAP            Product
	iscsi.1tb   ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    c7t0d0  ONLINE       0     0     0     1 TB           ST31000528AS
	    c7t1d0  ONLINE       0     0     0     1 TB           ST31000528AS
	    c7t2d0  ONLINE       0     0     0     1 TB           ST31000528AS
	    c7t3d0  ONLINE       0     0     0     1 TB           ST31000528AS
	  raidz1-1  ONLINE       0     0     0
	    c7t4d0  ONLINE       0     0     0     1 TB           ST31000528AS
	    c7t5d0  ONLINE       0     0     0     1 TB           ST31000528AS
	    c7t6d0  ONLINE       0     0     0     1 TB           ST31000528AS
	    c7t7d0  ONLINE       0     0     0     1 TB           ST31000528AS

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM     CAP            Product
	rpool       ONLINE       0     0     0
	  c3t2d0s0  ONLINE       0     0     0     160 GB         WDC WD1600BEKT-0

errors: No known data errors

  pool: storage_3tb
 state: ONLINE
  scan: resilvered 1.05T in 2h33m with 0 errors on Fri Dec  6 20:47:19 2013
config:

	NAME                       STATE     READ WRITE CKSUM     CAP            Product
	storage_3tb                ONLINE       0     0     0
	  raidz1-0                 ONLINE       0     0     0
	    c5t5000039FF4D13BDFd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13BFAd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13DCEd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13DD7d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13DDDd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13DEBd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	  raidz1-1                 ONLINE       0     0     0
	    c5t5000039FF4D13639d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13673d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D1367Cd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D1368Ed0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13A03d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4C4575Dd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	  raidz1-2                 ONLINE       0     0     0
	    c5t5000039FF4C3A938d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4C44847d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D11E06d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D12647d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4D13106d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4C477AFd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	  raidz1-3                 ONLINE       0     0     0
	    c5t5000039FF4C43E06d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ACA3
	    c5t5000039FF4CBFF77d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ABA3
	    c5t5000039FF4CBFFCCd0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ABA3
	    c5t5000039FF4CC0076d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ABA3
	    c5t5000039FF4CC01E8d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ABA3
	    c5t5000039FF4CC0284d0  ONLINE       0     0     0     3 TB           TOSHIBA DT01ABA3
	cache
	  c3t0d0                   ONLINE       0     0     0     240.1 GB       INTEL SSDSC2BW24
	spares
	  c5t5000039FF4D13623d0    AVAIL        3 TB           TOSHIBA DT01ACA3
	  c5t5000039FF4D13BDCd0    AVAIL        3 TB           TOSHIBA DT01ACA3



Code: 
 Pool 	 VER 	 RAW SIZE/ USABLE 	 ALLOC 	 RES 	 FRES 	 AVAIL zfs [df -h/df -H] 	 DEDUP 	 FAILM 	 EXP 	 REPL 	 ALT 	 GUID 	 HEALTH 	 SYNC 	 ENCRYPT 	 ACTION 	 ATIME 
 iscsi.1tb 	 - 	 7.25T/ 5.3TB 	 271K 	 - 	 - 	 5.34T   [5.4T /5.9T] 	 1.00x 	 wait 	 off 	 off 	 - 	 2658983613918912907 	 ONLINE 	 standard 	 n.a. 	 clear errors 	 - 
 rpool 	 - 	 149G/ 147.1GB 	 25.4G 	 - 	 - 	 107G   [107G /115G] 	 1.00x 	 wait 	 off 	 off 	 - 	 9276847531578541372 	 ONLINE 	 standard 	 n.a. 	 clear errors 	 off 
 storage_3tb 	 - 	 65T/ 52.1TB 	 30.3T 	 - 	 1.28T 	 26.9T   [27T /30T] 	 1.00x 	 wait 	 off 	 on 	 - 	 3243017313975357277 	 ONLINE 	 standard 	 n.a. 	 clear errors 	 -

12.11.2013-nas-7.jpg


I have a few more brackets and such coming in, not totally done yet.
 
Love the stack of stuff to the right lol

in the box some netgear stuff ( I think switches)
Laser printer in the background
Dell multimedia keyboard
Fans
HP laptop
Intel motherboard
Case for ram

It looks like your using multiple raid cards in that server what is connected to black mesh/ blueish silver cable that runs next to the black PSU on the left of the subsystem
I can see 2 of the cables going into some card.
 
Yeah I have a lot of crap, those are all asus rt-n16 routers. There is a bunch of ram in the case for ram.

The black/blue cable is an internal to external pass through bracket. I did clean up a bunch of the wiring since this pic and just need to get the drives mounted. There is a dell sas 6 card flashed to it mode that handles the drives in the lexan and an ibm m1115 crossflashed that is doing the shelves.
 
scotty, are you considering moving the system into a rackmount chassis? You already have 6u of JBODs so that's a nice reason to get yourself a rack :cool:
 
houkouonchi said:
Well this really sucks.. Someone broke into my house. They stole two laptops (one was my sisters I was fixing), a computer i used as a HTPC box, and both of my TV's (42 inch and 50 inch 4k one) as well as my semi-auto shutgun (with one box of ammo) as well as my nexus tablet.
...
Click to expand...

Screw the equipment. Another gun now in the streets. :rolleyes:
 
Concentric said:
scotty, are you considering moving the system into a rackmount chassis? You already have 6u of JBODs so that's a nice reason to get yourself a rack :cool:
Click to expand...

I've thought about it, I've looked at racks. I've looked at cases, the intel branded chassis are money.

ECC.Registered said:
Is that a E-ATX board in a ATX case?
Click to expand...

Yes. Nothing a dremel can't make fit.;)

2013-11-12%2014.05.07.jpg


2013-11-12%2015.35.37.jpg


I did get the bracket in for the floating drives.
 
Mozar said:
Very nice setup! So clean. I see you have a dual port NIC. Could you setup some kind of link aggregation and get 2Gbps on LAN? 10gb equipment still ain't cheap.
Click to expand...

pfft...! cheaper than ever nowadays.

2 of these http://www.ebay.com/itm/Mellanox-Infiniband-HCA-Single-4X-PCI-E-InfiniBand-Adapter-MHGS18-XTC-10Gb-CX4-/261317349968?pt=US_Internal_Network_Cards&hash=item3cd7ba5250

one of these http://www.ebay.com/itm/SAS-raid-cable-SFF-8470-CX4-Infiniband-to-SFF-8470-CX4-latch-type-10Gbps-100cm-/261298272082?pt=US_Drive_Cables_dapters&hash=item3cd6973752

about a hundred bucks
 
As an eBay Associate, HardForum may earn from qualifying purchases.
Only if you feel like dealing with infiniband, and the os you use supports it.
After a few months, I wrote infiniband off, as just too much of a pain to deal with, plus cables are short.
 
Re-did my file server last night, largely because it's now a bit ridiculous.
I used to have the whole "dual quad xeons w/HT + 48GB of ram + ZFS" so on so forth. Used a lot of power, and, seeing as how I was just storing movies on there, it largely went to waste. It could handle WAY more then I would ever use it for. So I swapped out the motherboard, and went to unRAID.
XabjJYol.png

The motherboard I went to is an AMD E-350 (1.6Ghz dual core) with 4GB of DDR3 1066.
Zb4Tkcul.jpg

Cable management is still a work in progress, soon as I find where I put my cache of zip-ties.

The Areca 1280ML that I'm using has a 1GB stick of ram, and no fan on the headsink, hence the case blower. Keeps it fairly cool
xFC20MRl.jpg


And finally, the drives: My cable management looks like hell, but there they are, twelve 2TB hard drives
xfrxeWtl.jpg


Full Load: 142W
Idle: 66W
 
Last edited:
What kind of software setup do you have on that system? I've wanted to configure a similar machine, but wanted ESXi running for VMs and some kind of raid setup that wasnt hardware. So I gave up and ended up with a Synology and a VM server. Though my board maxes at 24GB of ram :-(


sc0tty8 said:
I've thought about it, I've looked at racks. I've looked at cases, the intel branded chassis are money.



Yes. Nothing a dremel can't make fit.;)

http://sc0tty8.doesntexist.org/pics/systems/nas/2013-11-12%2014.05.07.jpg[../IMG]

[IMG]http://sc0tty8.doesntexist.org/pics/systems/nas/2013-11-12%2015.35.37.jpg[../IMG]

I did get the bracket in for the floating drives.[/QUOTE]
Click to expand...
 
cybertron said:
What kind of software setup do you have on that system? I've wanted to configure a similar machine, but wanted ESXi running for VMs and some kind of raid setup that wasnt hardware. So I gave up and ended up with a Synology and a VM server. Though my board maxes at 24GB of ram :-(
Click to expand...

Openidniana which is based on slowaris. There is a guide somewhere to put esxi on top. I am running vbox on it and am unable to get it to pass the cpu uage properly to the guest VM.

You can do phpvirtualbox.

Here is an AIO guide on esxi: http://www.servethehome.com/install-openindiana-esxi-50-allinone-zfs-server/

Here is a guide to add it to openindiana/napp-it: http://www.napp-it.org/extensions/phpvirtualbox.html
 
My current setup: Synology 1812+ with 8x 2TB w/ 10TB usable. Just making the cut for this thread.
Below that is my main rig, and an ESX 5.5 server I use for creating test cluster, etc.

69cf66f649aadcb63f487c529ceaab4e.png
 
Last edited:
sc0tty8 said:
Openidniana which is based on slowaris. There is a guide somewhere to put esxi on top. I am running vbox on it and am unable to get it to pass the cpu uage properly to the guest VM.

You can do phpvirtualbox.

Here is an AIO guide on esxi: http://www.servethehome.com/install-openindiana-esxi-50-allinone-zfs-server/

Here is a guide to add it to openindiana/napp-it: http://www.napp-it.org/extensions/phpvirtualbox.html
Click to expand...

Ah ok, thats some great info, thanks a bunch. Sadly my main concern is heat and so I'm trying to keep the cpu's to a minimum.
 
cybertron said:
My current setup: Synology 1812+ with 8x 2TB w/ 10TB usable. Just making the cut for this thread.
Below that is my main rig, and an ESX 5.5 server I use for creating test cluster, etc.
Click to expand...
The MikroTik router doesn't seem very useful with just one cable going in...
 
PigLover said:
The MikroTik router doesn't seem very useful with just one cable going in...
Click to expand...

It's not. I just picked it up and am transitioning from an RB450G which is behind the cable management block. I'm trying to to configure it quite differently this go around. Maybe some VM vlans, and a separate wifi network with limited IP access across the wall, etc.
 
Status
Not open for further replies.
Back
Top