The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
pjkenned said:
Scary. You aren't using all drives in one raid 4 array right?
Click to expand...

I am yes, but im not too worried about it. Any really critical data is already backed up elsewhere anyways. Plus if I lost more than 1 drive, you only lose the data that was on the drive that actually failed, as its not striped. So I think if you want a very large amount of storage at a low price, it's a very good system.
 
Updated with OS added (give Ender time to get it posted). I need OS for treadstone, houkouonchi, butcher9_9, d3vy, gjvrieze, MAFRI and n1ce_hat.

Thanks!
 
Total Advertised: 11TB
Total Available: 10.01TB

Specs:

Shuttle SG31G2
Intel C2D e8500
2gb (2x1gb) ddr2 800 of Crucial Ballistix Tracers
1x1tb Seagate Barracuda hd
2x1tb WD GP
4x2tb WD GP
2xSans Digital TR4M-B
LG GGC-H20L Bluray/HD-DVD drive

Here's an older pic, the only change is another TR4M-B added off to the side. :p

twAEm.jpg


And it's running WHS, which I've been very happy with.

w6G2G.jpg


Kinda wimpy compared to most here, but I'm glad I finally made the club. :D
 
longblock454 said:
Updated with OS added (give Ender time to get it posted). I need OS for treadstone, houkouonchi, butcher9_9, d3vy, gjvrieze, MAFRI and n1ce_hat.

Thanks!
Click to expand...

The System with the 5805 is running Windows Storage Server 2003 x64 R2 Unified.
The System with the 3220 is running Windows Server 2008 x64 Enterprise Edition.


MAFRI
 
Interesting, didn't even know I made the list :)
I was going to post here once I have the server in a usable state but it's cool to see the list is kept up to date.

My total is currently at 109TB (new media server at 100TB and my HTPC at 9TB) and not counting my other servers that will get removed from service once the new server comes online.

On the OS front, I haven't decided on that just yet, but it may look like it's going to be Windows Server 2008 R2. The HTPC is running Vista Premium 64bit right now although I have bought Windows 7 for it just haven't found the time to switch it yet... soon I hope :)

Hope to be able to post a picture of the server sometime within the next two weeks...
 
babyshark31 said:
Where'd you pick it up? Only place they mention on their website is NCIX and it's sold out apparently
Click to expand...

LightStar said:
Allready got 4 of them :p but thats in The Netherlands :+
Click to expand...

Yeah, I'm also in the Netherlands.

23tqosn.jpg


It's a really nice case for a 8disk + 1OS disk + 1 optical drive system. It comes with two silent 12cm fans included and you can add another 4 12/14cm fans if needed. Very solid and nice designed case for ~ €80,-.
 
moose517 said:
Amount of total storage 16TB
Amount of storage in the following system 16TB

Case - Norco 4020
PSU - antec basiqpower 450w
Motherboard - supermicro MBD-X7SBA
CPU - intel core 2 duo e6700
RAM - 4gb curcial 667 ECC ram
GPU (if discrete) - onboard
Controller Cards (if any) - 2x supermicro AOC-SAT2-MV8
Optical Drives - sony slimline
Hard Drives (include full model number) 7x WD 1TB black(WD1001FALS-OOJ7B), 2x WD 1TB geen(WD10EADS-00L5B1), 3x seagate 500GB(ST3500320AS), 2x seagate 500GB(ST3500630AS), Seagate 1.5TB(ST31500341AS), 2x hitachi 2TB(HDS722020ALA330)
Battery Backup Units (if any) - cyberpower 950w 1500va LCD version
Operating System - WHS

primary use is music/movie/picture storage. also have lots of undergoing programming projects stored on it. added lots of capacity this time around, hoping to get the rest of my DVD collection on it.
couple pics
Click to expand...

Moose, does the AOC-SAT2-MV8 support staggered spin up? I ask because 450W seems a little bit on the low side for that many drives...
 
MAFRI said:
So here I come :D
all system in the rack: 25,74TB
Click to expand...

I changed the config a bit:
Both 1HE Clients down there running a single 160Gb IDE and Win Server 2003 x86

The 2 HE Supermicro Storage Server with Server 2003 Storage x64 R2 have now 2x 160GB 2,5" @RAID1 for the OS@ICH9R and 6x1TB Seagate ES@RAID5@Adaptec 5805 and 2x300GB 15k SAS Seagate @RAID1@Adaptec 5805
and a 500GB Seagate as USB Drive for Backups.

The next Supermicro 2HE is running Server 2008 x64 EE
and have now 7x1,5TB Seagate@RAID6@Highpoint 3220 and one 750GB Seagate at the same controller. And one 1.5TB as USB Drive for Backups.

And the 4HE Supermicro Server have now 4x 250GB Seagate ES@RAID10@Adaptec 5805 and a single 1TB Seagate ES.

The Dell 2650 have 4x74GB SCA 10k Seagates @RAID5@Perc 3i QC.



#1


#2


#3


#4


MAFRI
 
12TB Advertised

Rosewill R101-P-BK MicroATX Case w/ 2x 120mm Scythe Kama Flow 2 @900rpm
Rosewill RG530-S12 530W
GIGABYTE GA-MA785GM-US2H
AMD Athlon II X2 240 @2.8GHz @1.175V
2 x 1GB Mushkin EPP DDR2 800
5 x 2TB SAMSUNG Spinpoint F3EG HD203WI
1 x 2TB SAMSUNG Spinpoint F3EG HD203WI in eSATA Enclosure
Testing out Vail. Will probably use WHSv1.

The motherboard/cpu/memory are from a retired HTPC setup. Not the sexiest case but fairly compact and fits 6 internal drives with minimal mods w/ room for expansion into 3x 5.25" bays. Despite over-provisioned PSU, the system is drawing just 47w idle, and only 35w with non-OS drives spun down. Consumption jumps to almost 90w with Prime95 and heavy disk activity. One huge surprise was a 10w drop in power consumption by simply installing most up-to-date drivers for video/audio/network even though Vail installed working drivers right outta the gate.

The system is replacing assorted external drives and a hot/noisy/slow 2TB Buffalo TeraStation that was using ~60w idle. WHS will primarily be used for media/file sharing and will be backing up 4 computers. External drive will probably be used for scheduled backups of OS and critical directories.

Hard Drives/Case:
img1855.jpg
img1856.jpg


Power consumption:
img1881.jpg
img1877.jpg


Disk Management:
vail.jpg
 
Last edited:
I will be going back up in the list in a bit =) Today I am picking up 20x2TB hitachi drives at fry's which should bring my total up to around 90TB. I might be upgrading my mythbox to 8x2tb as well which would get me over 103TB =)
 
houkouonchi said:
I will be going back up in the list in a bit =) Today I am picking up 20x2TB hitachi drives at fry's which should bring my total up to around 90TB. I might be upgrading my mythbox to 8x2tb as well which would get me over 103TB =)
Click to expand...

Pics or shens!
 
oh my
gratz on 20x, hope to see pics of hitachi packing, got just 5 for myself

is it a good to time announce i just got my first norco case? :D
 
jmroberts70 said:
Pics or shens!
Click to expand...

Here they are. Not a huge amount of packaging as these are OEM drives. First the shitty ass box that fry's put them in:



I figured I would get them in a box that holds 16 or 20 drives like we get them at work. If I would have known that was gonna happen I would have brought some of the supermicro boxes with me that I snagged from work that hold 16 drives each:



Here is 16 drives on top of my main norco rig:



And the last four added:



And the shot from the front of the machine.

 
And I started the migration process:

Code: 
root@sabayonx86-64: 06:51 AM :~# /root/cli64 vsf info
  # Name             Raid Name       Level   Capacity Ch/Id/Lun  State
===============================================================================
  1 WINDOWS VOLUME   18TB RAID6      Raid6     84.0GB 00/00/00   Rebuilding(4.9%)
  2 MAC VOLUME       18TB RAID6      Raid6     21.0GB 00/00/01   Need Rebuild
  3 LINUX VOLUME     18TB RAID6      Raid6   17895.0GB 00/00/02   Need Rebuild
===============================================================================
GuiErrMsg<0x00>: Success.

Code: 
root@sabayonx86-64: 06:51 AM :~# /root/cli64 disk info
  # Ch# ModelName                       Capacity  Usage
===============================================================================
  1  1  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  2  2  ST31000340AS                    1000.2GB  18TB RAID6
  3  3  ST31000340AS                    1000.2GB  18TB RAID6
  4  4  ST31000340AS                    1000.2GB  18TB RAID6
  5  5  ST31000340AS                    1000.2GB  18TB RAID6
  6  6  ST31000340AS                    1000.2GB  18TB RAID6
  7  7  ST31000340AS                    1000.2GB  18TB RAID6
  8  8  ST31000340AS                    1000.2GB  18TB RAID6
  9  9  ST31000340AS                    1000.2GB  18TB RAID6
 10 10  ST31000340AS                    1000.2GB  18TB RAID6
 11 11  ST31000340AS                    1000.2GB  18TB RAID6
 12 12  ST31000340AS                    1000.2GB  18TB RAID6
 13 13  ST31000340AS                    1000.2GB  18TB RAID6
 14 14  ST31000340AS                    1000.2GB  18TB RAID6
 15 15  ST31000340AS                    1000.2GB  18TB RAID6
 16 16  ST31000340AS                    1000.2GB  18TB RAID6
 17 17  ST31000340AS                    1000.2GB  18TB RAID6
 18 18  ST31000340AS                    1000.2GB  18TB RAID6
 19 19  ST31000340AS                    1000.2GB  18TB RAID6
 20 20  ST31000340AS                    1000.2GB  18TB RAID6
 21 21  N.A.                               0.0GB  N.A.
 22 22  N.A.                               0.0GB  N.A.
 23 23  N.A.                               0.0GB  N.A.
 24 24  N.A.                               0.0GB  N.A.
===============================================================================

Basically what I did was mount my main partition (18TB one with all my important shit) in read-only. I am going to leave it in read only until I one by one replace each drive with a 2TB drive which by my calculations will take about 8-9 days. I will be able to fall back on the old disks if anything happens during this process. Once everything has been moved over and stress tested a bit I will re-use the 1TB disks in an opensolaris box with ZFS and copy all my data to that and eventually re-create the arrays and file-systems on my main rig.

I don't trust jfs expansion as I need to do it right up to 32 TB (where the current utilities are limited at) and I wanted to allocate more space for my OS and windows slices as well.
 
drgnfang said:
Well at least you know when one goes the rest will follow shortly.
Click to expand...

Why do you say that? The serial numbers varied quite a bit on the drives and I don't know if you noticed but they weren't all from the same batch as a the manufacture date on them was not all the same.

And 5 drives down, 15 more to go:

Code: 
  # Ch# ModelName                       Capacity  Usage
===============================================================================
  1  1  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  2  2  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  3  3  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  4  4  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  5  5  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  6  6  ST31000340AS                    1000.2GB  18TB RAID6
  7  7  ST31000340AS                    1000.2GB  18TB RAID6
  8  8  ST31000340AS                    1000.2GB  18TB RAID6
  9  9  ST31000340AS                    1000.2GB  18TB RAID6
 10 10  ST31000340AS                    1000.2GB  18TB RAID6
 11 11  ST31000340AS                    1000.2GB  18TB RAID6
 12 12  ST31000340AS                    1000.2GB  18TB RAID6
 13 13  ST31000340AS                    1000.2GB  18TB RAID6
 14 14  ST31000340AS                    1000.2GB  18TB RAID6
 15 15  ST31000340AS                    1000.2GB  18TB RAID6
 16 16  ST31000340AS                    1000.2GB  18TB RAID6
 17 17  ST31000340AS                    1000.2GB  18TB RAID6
 18 18  ST31000340AS                    1000.2GB  18TB RAID6
 19 19  ST31000340AS                    1000.2GB  18TB RAID6
 20 20  ST31000340AS                    1000.2GB  18TB RAID6
 21 21  N.A.                               0.0GB  N.A.
 22 22  N.A.                               0.0GB  N.A.
 23 23  N.A.                               0.0GB  N.A.
 24 24  N.A.                               0.0GB  N.A.
===============================================================================
GuiErrMsg<0x00>: Success.

So far no timeouts, read errors, or re-allocated sectors on any of the drives.
 
Jesus. I take it the general consensus is that ZFS is godly and that I should learn it asap?
 
saedrin said:
Jesus. I take it the general consensus is that ZFS is godly and that I should learn it asap?
Click to expand...

The way I see it is that ZFS is one of several amazing options. ZFS is kind of at the tip of the blade when it comes to filesystem tech but there are a ton of really sweet choices out there right now. If you're doing this for home use, pick what works best for you. I'm going with a ZFS Server for online backups/storage and a 6x1TB intel RAID5 in my windows 7 workstation. for photo/video editing.
 
I'm running a simple 4x1.5TB on a PERC5 and it seems to be going okay, thinking about going to a norco soon, something expandable as my little array is completely full. Time to do some research, this thread seems to be a wealth of knowledge.
 
my little file server

media%20server%20003%20(Large).jpg


media%20server%20002%20(Large).jpg


media%20server%20001%20(Large).jpg


media%20server%20004%20(Large).jpg


the Dell PowerEdge 2900 is as follows

2x 80gb Velociraptors RAID 1 on Perc 5/i
8x 1Tb Seagate Barracuda 7200.12 RAID5+hotspare on Perc 5/i

Infortrend EonStor DAS (via FibreChannel)

16x 2Tb WD Green 32Mb cache drives TLDR disabled split into two RAID5+Hotspare arrays (top 8 and bottom 8) array 0 is assigned to FC port 0, and array 1 is assigned to FC port 1 for MOAR bandwith

theoretical storage in the dell is 8.08Tb das is 32Tb... so total of 40.08Tb
total actual storage in the dell is 6.08Tb das is, 24Tb so total is 30.08Tb
 
FLECOM said:
my little file server
Click to expand...
"Little" being the understatement of the year... lol

How a 2-case, 9U (or something on those whereabouts) setup can be called "little" baffles me. :p

Little or not, it does seem like IT p0rn, with that FC link keeping disk transfer speeds similar as those expected to be present inside the main case...

Thumbs up, and congrats on the build!

Miguel
 
awesome looking setup..... i know it is cool when i do not recognize the hardware :D
 
lol thanks... the poweredge 2900 is 5U's high...

it has a quad core xeon (think like 2ghz?) and 4gb ram... running win2k3 server (with COA on the side, got to be 100% legal!) raid is on a PERC5/i with 256mb ram and BBU... it doesn't really do much besides serve files via shares for my HTPC and an opensource PS3 media server app for my roommates so its more than powerful enough

FC card is an LSI PCIe 4x card...

the EonStor is I think 4U's high... has dual redundant PSU's (read: loud) and a single FC controller with 256Mb cache

the whole setup is on a SmartUPS3000RM at the bottom of the rack

sure its not the fastest/biggest but I just want something reliable and the PowerEdge 2900 and the EonStor's have been super good to me so I figure if I put them together, I should have a happy file server :)
 
Last edited:
Well I finally got all 20 of my disks replaced with hitachi:

Code: 
root@sabayonx86-64: 02:10 PM :~# /root/cli64 disk info
  # Ch# ModelName                       Capacity  Usage
===============================================================================
  1  1  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  2  2  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  3  3  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  4  4  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  5  5  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  6  6  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  7  7  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  8  8  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
  9  9  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 10 10  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 11 11  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 12 12  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 13 13  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 14 14  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 15 15  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 16 16  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 17 17  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 18 18  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 19 19  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 20 20  Hitachi HDS722020ALA330         2000.4GB  18TB RAID6
 21 21  N.A.                               0.0GB  N.A.
 22 22  N.A.                               0.0GB  N.A.
 23 23  N.A.                               0.0GB  N.A.
 24 24  N.A.                               0.0GB  N.A.
===============================================================================
GuiErrMsg<0x00>: Success.

I used the rescue option of:

RESETCAPACITY 18TB RAID6

To recalculate the capacity of the raid and added a new volume set to stress test the rest of the disks capacity and now my raid set shows its 40TB. Unfortunately it doesn't appear to give me a way to rename the raid set =P

Code: 
root@sabayonx86-64: 02:17 PM :~# /root/cli64 rsf info raid=1
Raid Set Information
===========================================
Raid Set Name        : 18TB RAID6
Member Disks         : 20
Total Raw Capacity   : 40000.0GB
Free Raw Capacity    : 0.0GB
Min Member Disk Size : 2000.0GB
Raid Set State       : Initializing
===========================================
GuiErrMsg<0x00>: Success.

I am currently intializing the new test volume:
Code: 
root@sabayonx86-64: 02:20 PM :~# /root/cli64 vsf info
  # Name             Raid Name       Level   Capacity Ch/Id/Lun  State
===============================================================================
  1 WINDOWS VOLUME   18TB RAID6      Raid6     84.0GB 00/00/00   Normal
  2 MAC VOLUME       18TB RAID6      Raid6     21.0GB 00/00/01   Normal
  3 LINUX VOLUME     18TB RAID6      Raid6   17895.0GB 00/00/02   Normal
  4 TEST VOLUME      18TB RAID6      Raid6   18000.0GB 00/00/03   Initializing(2.2%)
===============================================================================
GuiErrMsg<0x00>: Success.

So far there were no problems when replacing each disk one at a time. I had a directory with several hundred GB of video files and I did a md5sum before the migration and one after and the md5sum's all matched so I don't believe there was any corruption. The temps on the hitachi's are about the same as seagate:

Code: 
root@sabayonx86-64: 02:20 PM :~# /root/cli64 hw info
The Hardware Monitor Information
===========================================
Fan#1 Speed (RPM)   : N.A.
Battery Status      : 100%
CPU Temperature     : 46 (Celsius)
Ctrl Temperature    : 36 (Celsius)
Power +12V          : 11.916
Power +5V           : 4.999
Power +3.3V         : 3.296
SATA PHY +2.5V      : 2.496
DDR-II +1.8V        : 1.840
PCI-E +1.8V         : 1.824
CPU +1.8V           : 1.824
CPU +1.2V           : 1.200
DDR-II +0.9V        : 0.912
HDD #1  Temp.       : 37
HDD #2  Temp.       : 37
HDD #3  Temp.       : 38
HDD #4  Temp.       : 36
HDD #5  Temp.       : 39
HDD #6  Temp.       : 42
HDD #7  Temp.       : 42
HDD #8  Temp.       : 39
HDD #9  Temp.       : 41
HDD #10 Temp.       : 43
HDD #11 Temp.       : 43
HDD #12 Temp.       : 41
HDD #13 Temp.       : 41
HDD #14 Temp.       : 43
HDD #15 Temp.       : 44
HDD #16 Temp.       : 41
HDD #17 Temp.       : 40
HDD #18 Temp.       : 42
HDD #19 Temp.       : 42
HDD #20 Temp.       : 40
HDD #21 Temp.       : 0
HDD #22 Temp.       : 0
HDD #23 Temp.       : 0
HDD #24 Temp.       : 0
===========================================
GuiErrMsg<0x00>: Success.

I did notice that I seem to get more stable reads off the hitachi disks compared to the seagate ones. A dd read check went right @ 830 MB/sec:

Code: 
root@sabayonx86-64: 01:54 PM :~# dd bs=1M count=40000 if=/dev/sdc3 of=/dev/null
40000+0 records in
40000+0 records out
41943040000 bytes (42 GB) copied, 50.4663 s, 831 MB/s
 
Will be up by Friday. I'm travelling internationally for the weekened. So brain cycles are focused on tropical beaches :)
 
The backup sits in another building although there is only a single GigE line between the two.

The videos are only somewhat crucial though. We tape clients to prepare them for taking the stand and give them advice on things like demeanor, saying "uh... uh... uh...", shifting non-stop, not being surprised by litigator antics, and etc. Not overly critical since this is not like a deposition video or anything that would find its way into evidence.
 
I am definitely seeing some nicer speeds on my array since switching to hitachi drives. I created a test volume to stress test the other half of the drives and here is the read speed I got when reading the entire 18TB raid slice:

Code: 
17166132+0 records in
17166132+0 records out
17999994028032 bytes (18 TB) copied, 21999.1 s, 818 MB/s

Which is quite good and was consistent over the entire array. I have noticed reads seem much more consistent and don't drop down at all like they did with my seagate disks. Here is what I got for writes:

Code: 
dd: writing `/dev/sdd': No space left on device
17166117+34 records in
17166116+34 records out
17999994028032 bytes (18 TB) copied, 22649.6 s, 795 MB/s

Which I was pretty impressed with. Write speeds are only *slightly* slower than reads on my ARC-1280ML in raid6 on these hitachi drives. Almost broke 800 MB/sec writes!
 
houkouonchi said:
I am definitely seeing some nicer speeds on my array since switching to hitachi drives. I created a test volume to stress test the other half of the drives and here is the read speed I got when reading the entire 18TB raid slice:

Code: 
17166132+0 records in
17166132+0 records out
17999994028032 bytes (18 TB) copied, 21999.1 s, 818 MB/s

Which is quite good and was consistent over the entire array. I have noticed reads seem much more consistent and don't drop down at all like they did with my seagate disks. Here is what I got for writes:

Code: 
dd: writing `/dev/sdd': No space left on device
17166117+34 records in
17166116+34 records out
17999994028032 bytes (18 TB) copied, 22649.6 s, 795 MB/s

Which I was pretty impressed with. Write speeds are only *slightly* slower than reads on my ARC-1280ML in raid6 on these hitachi drives. Almost broke 800 MB/sec writes!
Click to expand...
Very impressive.
 
Last edited:
Status
Not open for further replies.
Back
Top