The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
Blue Fox: PM. i am in need of a 1280ML and have a brand spankin new (besides trying to get 1.5tb drives working on it) 1680ix-24
 
Total Advertised: 20.65 TB
Total Available: 17.88 TB

Update - 2010-02-27: Added more drives (broke 20TB advertised!), finally have pics of the sytem.
Update - 2009-08-01: Replaced the internal 750g drive with a new 2tb drive. Replaced external 750g (died) with 500g. New totals: 11.51 tb/2.4 tb. Still no pictures of the system.

Case: Norco 4020
PSU: PC Power and Cooling Silencer 750 Red (haven't tested with kill-a-watt since rebuild with new PSU)
Mobo: Gigabyte GA-EP45-DQ6 (Shipped with F12 Bios for whoever asked this earlier)
CPU: E7200
HS/F: Thermal Right Ultima-90/Scythe 92mm Kama PWM
RAM: G.Skill 4x1g DDR2/667
GPU: Jaton GeForce 6200 PCI
OS Drive: WD RE3 250g
Controller #1: Supermicro AOC-SASLP-MV8
Internal Drives: 5x WD 2tb (all EADS), 7x WD 1tb (2x EACS, 5x EADS)
External Drives:
  • AMS T4U with 2x500g and 2x200g (IDE) 1.4TB Advertised, 1.28TB Available
  • Vantec Nexstar MX-SR with 2x500g

Fans:
I replaced the two rear fans with Arctic Cooling F8 PWM (they didn't have any non-PWM) 80mm fans. These run at ~ 2000rpm. I replaced the five fans in the middle with a mix of Scythe S-Flex 80mm fans. The two outside fans are 2800rpm and have been moved to the other side of the assembly. The three inner fans are 2000rpm. All fans, except the CPU, are hooked up via a PCIe 6-pin -> Molex converter. Overall, I am happy with the noise/cooling ratio of these fans. The S-Flex 2800rpm have a nasty high-pitch to them, but are otherwise acceptable. I may look at getting some noise absorption material to put in various parts of the case.

vault-console-001.jpg

vault-console-002.jpg


Updated: System Pics! (ignore the bottom HBA, older SM AOC-SAT2, it has been removed)
vault-001.jpg

vault-002.jpg

vault-003.jpg

vault-004.jpg

vault-005.jpg
 
Last edited:
drgnfang said:
So, I finally finished the rebuild of my WHS. Things to do: Add second controller card, get more drives. Get a real switch so I can team the 4 NIC's on the mobo.

For the moment I have the following:
Total Advertised: 10.6 TB
Total Available: 4.28 TB

Case: Norco 4020
PSU: Corsair HX520 (current max usage @ spinup is 200w, not sure about amp)
Mobo: Gigabyte GA-EP45-DQ6 (Shipped with F12 Bios for whoever asked this earlier)
CPU: E7200
HS/F: Thermal Right Ultima-90/Scythe 92mm Kama PWM
RAM: G.Skill 4x1g DDR2/667
GPU: Jaton GeForce 6200 PCI
OS Drive: Seagate 7200.10 250g
Controller #1: Supermicro AOC-SASLP-MV8
Internal Drives: 7x WD 1tb, 1x WD 750g
External Drives:
  • AMS T4U with 2x500g and 2x200g (IDE) 1.28tb advertised
  • Vantec Nexstar MX-SR with 2x500g
  • AMS DS with 750g
  • Macally with 500g

Right now all of the internal drives are connected to the supermicro card. I can add four more drives (5 if I don't mind removing it later) before I have to get the second controller card. One bit of weirdness is the drives hooked up to the SM card show up as "unknown" for location.

Fans:
I replaced the two rear fans with Arctic Cooling F8 PWM (they didn't have any non-PWM) 80mm fans. These run at ~ 2000rpm. I replaced the five fans in the middle with a mix of Scythe S-Flex 80mm fans. The two outside fans are 2800rpm and have been moved to the other side of the assembly. The three inner fans are 2000rpm. All fans, except the CPU, are hooked up via a PCIe 6-pin -> Molex converter. Overall, I am happy with the noise/cooling ratio of these fans. The S-Flex 2800rpm have a nasty high-pitch to them, but are otherwise acceptable. I may look at getting some noise absorption material to put in various parts of the case.

Pictures of the server will come this weekend, until then here are WHS console images.
s-console-001.png

hhs-console-002.png


On a side note, I moved the drives around and "lost" their places for the wireframe - does anyone know of a add-on that lets you "spin up" the activity light so you can ID a drive location?
Click to expand...


Pull a drive and map its lost location.
 
Main Server: Supermicro 846TQ with 2x 3.2Ghz Xeon, 8Gb Ram, 74Gb Raptor, 20x 1Tb Hitachi & 4x500GB WD drives on a 3ware 9650-24M8, QLogic 4010C iSCSI HBA. (22Tb Advertised, 18.625Tb Useable)

SAN: Infortrend iSCSI-SATA Raid-6 System with 16x 1Tb Hitachi Drives (16Tb Advertised, 13Tb Usable) connected directly to the QLogic 4010C HBA.

Total Storage: 20Gb+2Gb & 16Gb = 38Gb
Usable Storage: 16,763Gb+1,862Gb & 13,037Gb+ = 31,662Gb (or 31Tb)

Some pics:
screen1.jpg

screen2.jpg

tq8461.jpg

infortrend1.jpg

infortrend2.jpg
 
[LYL]Homer;1034269130 said:
Updated my server post with Kill A Watt results. Anyone else have some wattages?
Click to expand...

Incredibly low power use for so many drives.

I dunno how you do it.

My rig pulls about 300w average, 270w idle/325w load.

10580GB Advertised

Norco 4020 Rev.3
PCP&C Silencer 750 Crossfire
ASUS M2N32-SLI DLX Wi-fi
AMD Phenom X4 9150e 65w 1.8GHz w/ Xigmatek HDT-SD964
2x2GB Kingson DDR2-800
EVGA 7800GT 256MB
Supermicro AOC-SASLP-MV8
Supermicro AOC-SAT2-MV8
Non-Pool drives:
  • Maxtor 320GB (Seagate 7200.10) IDE for OS
  • 2xSeagate 320GB 7200.10
Pool drives:
  • 1x2TB WD Green
  • 4x750GB Hitachi
  • 1x750GB WD Black
  • 1x750GB Seagate 7200.12
  • 2x500GB Samsung F1
  • 1x500GB Seagate 7200.10
  • 1x500GB Seagate 7200.9
  • 1x400GB Seagate ES
  • 1x400GB Seagate 7200.10
  • 1x320GB Seagate 5400.5
7x80mm Zalman case fans
2x92mm Arctic Cooling PWM for CPU
2x Thermaltake PCI Slot Blowers
1x120mm Lian-Li Exhaust
 
I'll update my post in a bit, but I'm getting about ~ 120 - 130w idle, a max spike of ~200w at full boot spin-up, and ~ 130 - 140w when transferring data both ways. That's with a total of 9 drives internal.

thebeephaha - Try replacing all those older/smaller drives with new WD green drives. That alone should drop your idle. Also, consider going with a smaller PSU (I LOVE the 750 silencer, but you're not using anywhere near 750w) or a PSU that has higher efficiency at lower watts (like the Antec signature, or the new Corsair 850HX). Make sure your system is set to use lower clock/multi when idle. Might also look at swapping out the Quad for a dual core, unless you really need the quad.
 
Some good ideas. I did just take out 7x320GB drives and replaced it with the one 2TB so it is a start.

I know my video card is a hog but it's being severely underclocked, at the time it was all I had spare to use.

I'm gonna keep the quad, its a pretty low power entry level quad using less than my previous X2 7750 dual core. Besides I use my server for a lot of different things, the quad comes in handy.

Never thought about the power supply... Good idea.
 
I'm getting around 260-270 idle and around 370-390 under load. This is my current setup with 20x 1TB ES.2's.
 
thebeephaha said:
Incredibly low power use for so many drives.

I dunno how you do it.

My rig pulls about 300w average, 270w idle/325w load.
Click to expand...

Hmmm, I'm curious too, so I looked up some numbers.

I only have 3x 120mm fans (not including PSU and HSF fans) whereas you have 12 fans, it looks like typical 60-120mm fans pull anywhere from 2.5-6.6 watts. Difference of 21-71 watts depending on fans.

No video card in mine, looks like the 7800GT at idle is about 20 watts.

Does the Wi-Fi use a lot of energy? Maybe turn it off and test?

We both have 65w cpus, may be a difference in idle wattage.

You have 17 drives, I have 11.

Here's a breakdown of wattages for mine:
Code: 
read/write  idle    drive                  notes
5.4          2.8    WD10EACS  (qty=5)      1tb green
5.4          2.8    WD7500AACS  (qty=1)    750gb green
8.8          8.4    WD7500AAKS  (qty=2)    750gb 7200rpm
10.1         8.3    ST31500341AS  (qty=2)  1.5tb 7200.11
14.0        10.0    7H500F0  (qty=1)       OS, 500gb Maxtor
=========
84.2 load /  60.2 idle  = total for all drives


Looks like your post on your config is being updated so I can't see what the model of all your drives are to look up. Maybe you can look them up and post so we can compare.
 
[LYL]Homer;1034275054 said:
Hmmm, I'm curious too, so I looked up some numbers.

I only have 3x 120mm fans (not including PSU and HSF fans) whereas you have 12 fans, it looks like typical 60-120mm fans pull anywhere from 2.5-6.6 watts. Difference of 21-71 watts depending on fans.

No video card in mine, looks like the 7800GT at idle is about 20 watts.

Does the Wi-Fi use a lot of energy? Maybe turn it off and test?

We both have 65w cpus, may be a difference in idle wattage.

You have 17 drives, I have 11.
Click to expand...

I thought about fans, though I have them all pretty much undervolted by the motherboard thanks to ample fan headers controlled by ASUS Q-Fan.

I disabled the Wi-Fi and the second LAN, also since the video card is underclocked it may use less even.

Yep, 65w quad is nice, and I have Cool and Quiet on so idle the CPU clocks down to 900MHz vs 1.8GHz.

I'm guessing the difference is caused by a number of things as mentioned above but it may be the sheer fact my motherboard itself uses more power than yours does. NVIDIA chipsets are not always the most efficient, especially an "old" 590 SLI.
 
I'd be willing to bet that a lot of it is on your hdd's.

The greens idle at 2.8, everything else I have is 8.3 or higher at idle.
 
[LYL]Homer;1034276713 said:
I'd be willing to bet that a lot of it is on your hdd's.

The greens idle at 2.8, everything else I have is 8.3 or higher at idle.
Click to expand...

I did some tweaking for power settings, got it down to around 225w average. AMD's Cool and Quiet is a wonderful thing.

If I had green drives I guess I would hit about what you do then.
 
120-160w on idle, up to 160-170w on full load, normal day2day use when all except sys are spindowned: 40w
dunno about quad boxes, have them coming home tomorrow, will see
 
Should have something interesting coming in the mail tomorrow. No pics until then, but it involves a 4U Supermicro case. :D
 
i'm in the same boat hehe. can't sell drives for about a month (want to give the RAID a month to prove stable)
 
Don't rule out the possibility of an impending disk failure, no matter how cool or how little it is accessed.
 
bump
seems like summer and heat has made this topic pretty slow
lets heat it up with slight updates: 53 -> 59 :D
third rebuild coming this night
 
sorry for double,
but finished the updates, now i rest at @ 72 ... Ockie, im waiting for hostile overtaking
 
Last edited:
asgards said:
sorry for double,
but finished the updates, now i restat @ 72 ... Ockie, im waiting for hostile overtaking
Click to expand...

I'm enjoying the summer, lol. I cut back quite a bit with the new drive upgrades, they are so much nicer... less than half the drives for the same space.
 
Im also pretty quite aswell, a few new drives but not intrestin enough to update
 
Amount of total storage - Advertised: 11TB Actual: 10.49TB
Amount of storage in the following system - Advertised: 11TB Actual: 10.49TB

Case: Norco 4020
PSU: Enermax EMD425AWT MODU82+ 425W
Motherboard: Asus K8N4E deluxe
CPU: AMD Sempron 2800+ (socket 754 1.6GHz)
RAM: 512MB DDR 1
GPU: PCI
Optical Drives: None
Controller Cards: 1x Dell Perc 5/i
Hard Drives: 4x Seagate 1.5TB 7200rpm, 3x Seagate LP 1.5TB 5900rpm, 1x WD 5000AAKS 500GB
Operating System: Arch Linux on a 4GB USB stick.

Goals of this system:
As cheap as possible bulk storage, with minimal power consumption and noise, as i have to sleep in the same room as the server. Also, upgrading disks should be as hassle-free as possible

front view
sc02659resize.jpg


top view
sc02655resize.jpg


As you can see, the limited length of the fan cables, makes cable management very difficult.

Cooling: I have modded the Norco 4020 case with 3x Coolermaster R4-S4S-10AK 140mm (1000rpm) fans in the middle, instead of the 5 screaming 80mm fans it came with (I also removed the mid panel). I have also changed the 2 80mm fans in the back with 2 Nexus 80mm fans @ 1500rpm. The CPU is cooled with a Coolermaster RR-CCH-PBU1-GP GEMINII S cooler, also with a 120mm fan.

overview of 140mm fans
sc02654resize.jpg


140mm fan detail
sc02658resize.jpg


As you can see, the 140mm fans sit on top of a folded bubble-wrap envelope, with naked bubble-wrap on top of them. This is a quick solution that works very well, with very little air leakage.

The system is very silent, and i can sleep easily with it being about 2 meters away from my bed. Disk temps are around 30 degrees Celsius for the Barracuda LP drives and 36 for the 7200rpm ones (with my house at around 25 degrees).

I do not use RAID, because I change my drives very often as i upgrade to the newest sweet spot capacity and raid would be a PITA. Every disk is exported via the gigabit lan as a Samba share. My OS is Arch Linux, without X, just console mode for me. It is installed on a ext2 formatted 4GB USB stick, that makes it very easy to backup and much more silent than a small hard drive would be, not to mention the fact that i dont have to dedicate a SATA port for it.

The motherboard comes with 8 SATA ports, and i have bought off ebay 3 PERC 5/i controllers as they are incredibly cheap for the number of ports they provide. If you use them without raid5 (unfortunately they do not expose non raided drives to the OS, so i have to make a RAID-0 for every drive), they do not need any extra cooling. The most basic drawback is that i do not have SMART monitoring on the disks that are on the controller.

back of the case
sc02656resize.jpg


I was a big fan of the Western Digital WD10EACS drives (I had 10 of them), as they were dead silent and very cool. So I was surprised to find that the Barracuda LP 5900rpm are noticeably cooler and just as silent! Of course the 7200rpm Seagates are hotter and somewhat audible when seeking, so they will be the first ones to go on the next upgrade. The 500GB drive is left on the system for backups, or for intermediate space when moving stuff around.

I have consolidated all my storage into this box, movies, music etc, and i also backup my main system on it. The Norco case was a pain to get to Greece, but worth every penny of the expenses i made for it.

List of disks on the system (the 3 Perc 5/i lines are duplicates)
listdisks.png


Finally, hats off to Ockie and to the rest of the people here, their incredible builds are truly an inspiration.
 
Last edited:
Hi guys I am about to put this together and wondering if anyone had any comments or suggestions:

Case: Norco 4220
Controllers: Perc 5/i + BBU + SAS->SATA adapter cable
OS: ESXi running Windows Home Server VM
HDD 8 x 750
Memory: 10GB

Will add another Perc 5/i + 8 1TB drives asap.

Now assuming the above setup will running the WHS in a VM cause any problems? I will not be using RAID as I realise WHS doesn't like that.

The mobo is a supermicro board and I have 2x 2214 Opteron CPUs, the only issue with my current server is it is LOUD. Wondering if anyone could give me any tips on how to silence this beast? The mobo allows a server and workstation fan speed config and I have it set on workstation. The 2 HSFs are quite loud but its hard for me to find other HSF that fit and that are quiet at the same time.

Any issue with running the Perc 5/i in 8x PCI-E slot compared to 16x?

EDIT: Im actually thinking about selling the CPUS, HSF and Motherboard and getting something more inline with a workstation mobo/cpu combo. Although I plan to run a few VMs I am not sure but surely a nice quad core CPU will suffice? but then of course it has to be compatible with ESXi....
 
Last edited:
Status
Not open for further replies.
Back
Top