The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
Killer Chris said:
Newest addition:

EMC² CLARiiON AX150 /w 12x 500GB HDDs :)

main.php
Click to expand...

Connect the second power outlet too ;). In some occasions write cache gets disables when only 1 PSU is connected. Also, Uni/Navisphere is a big nag is something isn't plugged correctly.
 
Here's the link to my build thread: http://hardforum.com/showthread.php?t=1656966

Purpose of this server is to store all of my media and stream it to media centers around my house.

-Specs-
Case: Norco 4220
CPU: Intel Core i3 2105
Motherboard: BIOSTAR TH67B
RAM: 8gigs DDR 3 1600 Kingston HyperX
RAID Controller: LSI MegaRAID SAS 9260-16i
Hard Drives: 1x 1TB (OS), 7x 2TB (RAID 5 Storage Pool)
Power Supply: Corsair 750W

And here's a couple pics and a video for you guys:

http://youtu.be/Qj1Sk2sllcA?hd=1

IMG_0109.JPG


IMG_0151.JPG
 
Does a server benefit much from an SSD for the OS drive? I mean, doesn't like boot once and do everything in RAM after...
 
This is something I have wondered also, the only area where SSD's would really benefit in servers is in database/application servers or as scratch disks/caches. I don't see much point having an SSD for an OS drive personally.
 
I also was thinking to use an SSD in my fileserver . but it makes no sense for me. Ended up with a 320GB 7200RPM 2.5inch bootdrive .
 
I was going to use USB sticks as boot drives for my Xen pool but then I didn't have any so I just used laptop drives instead...
 
How do you backup your host systems when they are on a usb ?

was going to do the same, but choose to use an Raid 10 system with 4x500gb sas, my vm datastore is on there
 
Sorry for the lack of updates. I moved, work etc. If your update/addition wasn't included make sure you PM me with the most total and most in a single chassis specifically outlined.

Thanks
 
I recently updated my configuration. I'm now running ESXi to virtualize my router, TV recording box, and file server in one piece of hardware (and saving 100+ watts over my last setup!).

Total storage (all in one box): 24.113TB

Hardware list:
  • Supermicro H8DCL-iF (6F was an option, but would have needed fan-in cables to work, which is icky)
  • 16GB DDR3 ECC (for now, 32GB coming up one of these days)
  • Opteron 4170, possible upgrade to 4234 later. 6 cores, 2.1 GHz, 50W TDP.
  • VMware ESXi 5.0
  • 32GB SSD as boot disk/datastore for ESXi
  • 6 5k3000 3TB drives, main data store (raidz2)
  • 6 7k1000 1TB drives, backups (raidz3)
  • USB3 controller for PCI passthrough (AVR experiments)
  • Supermicro AOC-USAS2-L8i LSI SAS2008 controller, passthrough to the file server
  • HP SAS expander (awesome piece of kit)
  • Intel 6300 AGN wireless card, passthrough to firewall
  • Ceton InfiniTV card, passthrough to Linux box (since the Windows VM apparently won't deal with it directly)
  • Enermax ERV1050EWT Revolution85+ 1050W modular power supply
  • Norco RPC-4220
  • ANS-9010B ram drive for ZIL, 1GB ECC ram installed
  • 80GB laptop drive from old machine, so I can retrieve anything I need from it

I'm currently running 4 production VMs: one OpenIndiana for storage and file serving (which serves an NFS share back to the ESXi host), one PFsense for routing my Internet connection, one Linux for handling the Ceton card and the USB/AVR experiments, and a Windows 7 box for running the vSphere client over remote desktop and WMC.

I have 12TB more storage (6 2TB drives) sitting on the shelf; I don't need it right now, and it costs power to run them for no reason.
 
unhappy_mage said:
I recently updated my configuration. I'm now running ESXi to virtualize my router, TV recording box, and file server in one piece of hardware (and saving 100+ watts over my last setup!).

Total storage (all in one box): 24.113TB

Hardware list:
  • Supermicro H8DCL-iF (6F was an option, but would have needed fan-in cables to work, which is icky)
  • 16GB DDR3 ECC (for now, 32GB coming up one of these days)
  • Opteron 4170, possible upgrade to 4234 later. 6 cores, 2.1 GHz, 50W TDP.
  • VMware ESXi 5.0
  • 32GB SSD as boot disk/datastore for ESXi
  • 6 5k3000 3TB drives, main data store (raidz2)
  • 6 7k1000 1TB drives, backups (raidz3)
  • USB3 controller for PCI passthrough (AVR experiments)
  • Supermicro AOC-USAS2-L8i LSI SAS2008 controller, passthrough to the file server
  • HP SAS expander (awesome piece of kit)
  • Intel 6300 AGN wireless card, passthrough to firewall
  • Ceton InfiniTV card, passthrough to Linux box (since the Windows VM apparently won't deal with it directly)
  • Enermax ERV1050EWT Revolution85+ 1050W modular power supply
  • Norco RPC-4220
  • ANS-9010B ram drive for ZIL, 1GB ECC ram installed
  • 80GB laptop drive from old machine, so I can retrieve anything I need from it

I'm currently running 4 production VMs: one OpenIndiana for storage and file serving (which serves an NFS share back to the ESXi host), one PFsense for routing my Internet connection, one Linux for handling the Ceton card and the USB/AVR experiments, and a Windows 7 box for running the vSphere client over remote desktop and WMC.

I have 12TB more storage (6 2TB drives) sitting on the shelf; I don't need it right now, and it costs power to run them for no reason.
Click to expand...


Pictures????
 
CubanBlood said:
Pictures????
Click to expand...

My camera isn't working right now (I put ICS on my phone, and there aren't any drivers for the camera yet :() but I can borrow one when I get home and take some pics. In the meantime, here's some screenshots from the console/whatever.
Code: 
will@thor:~$ zpool list
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
backup  5.44T  2.09T  3.35T    38%  1.00x  ONLINE  -
huge    16.2T  10.6T  5.66T    65%  1.00x  ONLINE  -
rpool   15.9G  3.05G  12.8G    19%  1.00x  ONLINE  -
Here we have the 3TB backup pool (zpool list shows total capacity of the disks, including the space that's used as parity), the 12TB main data pool ("huge"), and the 16GB ESXi boot disk.

esxi-summary.png

Here's the front page of the ESXi console.
 
Hello

please find underneath my new server based on PC Q25

CASE : Lian Li PC-Q25
PSU : BeQuiet 350W 80+ Gold
MB : Asus P8H67-I
CPU : Core I3-2100
RAM : Gskill DDR3 1600
STORAGE : 12.064 TB
- SSD OCZ 64 GB
- 2*2TB WD EARS
- 1*2TB WD EARX
- 2*2TB Samsung F3 ECO 2
- 1*2TB Seagate Green
PCIE : Sata/Esata 3 Card
OS : Server 2008 R2


- Video& music streaming
- UPNP
- FTPES
- …

Connected to gigabit lan and optical fiber 100/100MB
 
It's not much storage to brag about, but i thought i would post pictures of Inside & out of a QNAP 410u.

4 x 1tb drives @ 7200 with raid 5 & hotspare..

DSCN0153.JPG

DSCN0155.JPG

DSCN0156.JPG

DSCN0157.JPG
 
dashpuppy said:
It's not much storage to brag about, but i thought i would post pictures of Inside & out of a QNAP 410u.

4 x 1tb drives @ 7200 with raid 5 & hotspare..
Click to expand...


How loud are those little fans? I've wanted to get one of these for a long time, but the noise seems like it would kill the deal. I have used some Dell 860's that are pretty quiet, so hopefully this unit is smart enough to slow down. :)
 
cybertron said:
How loud are those little fans? I've wanted to get one of these for a long time, but the noise seems like it would kill the deal. I have used some Dell 860's that are pretty quiet, so hopefully this unit is smart enough to slow down. :)
Click to expand...

its dead silent :) when you power it on the fans run then after that they run at such low rp you never even hear them.. TRUST ME its very very very quiet..
 
Scharnhorst said:
Hello

please find underneath my new server based on PC Q25

CASE : Lian Li PC-Q25
PSU : BeQuiet 350W 80+ Gold
MB : Asus P8H67-I
CPU : Core I3-2100
RAM : Gskill DDR3 1600
STORAGE : 12.064 TB
- SSD OCZ 64 GB
- 2*2TB WD EARS
- 1*2TB WD EARX
- 2*2TB Samsung F3 ECO 2
- 1*2TB Seagate Green
PCIE : Sata/Esata 3 Card
OS : Server 2008 R2


- Video& music streaming
- UPNP
- FTPES
- …

Connected to gigabit lan and optical fiber 100/100MB
Click to expand...


nice :D i love mine - going to mod it so i can add two more drives
 
Hello

Fans provided in the bundle are not as silent as expected however I was a bit afraid to have some vibrations or resonance due the HDDs but fortunately no problem on that side.
 
Here is my little fileserver:


u2010_74.jpg



u2010_75.jpg



u2010_37.jpg



u2010_39.jpg



Intel D510MO Fanless Dual Core Atom Mini-ITX Board.
RAIDCore BC4852 RAID controller.
5 x 2TB Samsung HD203WI Spinpoint F3EG harddrives.
160 GB Samsung HM160HI SpinPoint M5 2.5" harddrive.
StarTech 2.5" SATA removable harddrive rack for PC expansion slot.
Samsung SN-T083C/BEBE slot loaded optical drive.

Full worklog here
 
Ah, another Eve player (and Veto at that)..

Haven't seen you guys around lately. You taking contracts these days?
 
For those with these 10+ TB systems, how much maintenance are they on average?

I mean, how often does a hard-drive die and require replacement? Is it hassle-free to just swap it out or remove it, or does it takes some time/effort to get things back to the point you don't have to keep doing something?
 
shurcooL said:
For those with these 10+ TB systems, how much maintenance are they on average?

I mean, how often does a hard-drive die and require replacement? Is it hassle-free to just swap it out or remove it, or does it takes some time/effort to get things back to the point you don't have to keep doing something?
Click to expand...

For me, little hassle.
Logon to server console and identify failed drive
Shut down server and remove failed drive
Remove failed drive from its caddy and replace with a fresh drive (i have spares)
Replace new drive
Reboot server
Start rebuild
Send failed drive back to seagate
 
CubanBlood said:
Do you have two harddrives just there, or are they attached to something?
Click to expand...

at one point i had 3x 80mm fans in the bottom of the case the the screw holes for the fans just happen to line up with the harddrive holes in their current position
 
Asposium said:
For me, little hassle.
Logon to server console and identify failed drive
Shut down server and remove failed drive
Remove failed drive from its caddy and replace with a fresh drive (i have spares)
Replace new drive
Reboot server
Start rebuild
Send failed drive back to seagate
Click to expand...
How do you find out that there's a failed drive? Does it let you know somehow on its own, or do you have to keep checking yourself manually?

How often, on average, do you have to deal with a failed drive? Once a year? Once a month?

Rpkole said:
at one point i had 3x 80mm fans in the bottom of the case the the screw holes for the fans just happen to line up with the harddrive holes in their current position
Click to expand...
I think he meant if it's connected to the computer via a data cable, or if it's sitting unconnected.
 
To find out about a failed drive there are some options...
-checking the management software
-mail notifications
-status-LEDs of SAS-backplanes
-alarm buzzers
-...
Often the controller card will also give warnings when a failure is predicted.

Its hard to give a number of failing drives a year - the more drives you use, the more could fail, then you may get drives of a bad batch,...
The worst case were 3 or 4 drives within 3 weeks, but then half a year nothing happened.

On the maintenance - this is quite minimal, with hot plug cages you can change the failed drive within seconds (well, if a spare is available) and the rebuild should start as the drive is pushed in.
But this can extend to hours and days if the failures get IT-like totally strange, a failing drive is one of the nicer things...
 
shurcooL said:
I think he meant if it's connected to the computer via a data cable, or if it's sitting unconnected.
Click to expand...

O lol its a raid 1 array for my OS

shurcooL said:
How do you find out that there's a failed drive? Does it let you know somehow on its own, or do you have to keep checking yourself manually?

How often, on average, do you have to deal with a failed drive? Once a year? Once a month?
Click to expand...

some raid cards will email you when something goes wrong with an array.

as for failed drives i have only had 1 died on me in the past 2 years *knocks on wood* and my fileserver see lots of action at LAN's
 
shurcooL said:
For those with these 10+ TB systems, how much maintenance are they on average?

I mean, how often does a hard-drive die and require replacement? Is it hassle-free to just swap it out or remove it, or does it takes some time/effort to get things back to the point you don't have to keep doing something?
Click to expand...

Today I spent a couple hours on maintainance. I upgraded my OS drive to a SSD. I needed an extra SATA port. Tried a PCIe SATAIII 2 port card I had lying around. Caused my TV recording software problems and had a conflict with my DVD playing software. 7 minutes to copy my OS to the SSD and 2 hours testing and recovering. Tried a PCIe SATAII 4 port card I had purchased. 7 minutes to copy the OS to the SSD and 5 minutes of testing.

No hard drive failures since I built the server - years ago. Just a few minutes to change drives after several hours of copying data from the old to the new. 16TB of storage (8 2TB drives) currently ; 2-3TB of free space. Ports for 3 more hard drives.

---

I have my hard drives hung from a 1/8" piece of plywood. They are held in place with 2 screws. Unplug, sunscrew. Screw, plug.
 
I have learned to not post anything that might be traced back to me.

But it is nothing fancy. A piece of plywood 5"x11" with 2 rows of holes. Each with 8 holes spaced 1-1/4" apart I think the holes are a little bit sloppy for 8-32 screws. (the screws go in the sides of the drives)
 
GeorgeHR said:
I have learned to not post anything that might be traced back to me.

But it is nothing fancy. A piece of plywood 5"x11" with 2 rows of holes. Each with 8 holes spaced 1-1/4" apart I think the holes are a little bit sloppy for 8-32 screws. (the screws go in the sides of the drives)
Click to expand...

I understand. Is it open on both fron and back? How is the heat level?
 
Status
Not open for further replies.
Back
Top