The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
I'll just post mine here:

Main PC (used for video processing, software development and gaming)
Case: Fractal Design Define R3 Black
CPU: i7 920 @ 3.4GHz with a Noctua NH-U12P SE2 cooler
Mobo: Asus P6T Deluxe V2
RAM: 12GB (3x4GB) of Crucial Ballistix DDR3-1600
Non-RAID storage: 256GB Crucial m4 for boot + applications - attached above the optical drives at the top of the chassis
RAID storage: 8x Hitachi Ultrastar A7K2000 - 2TB on a LSI 9260-8i RAID6 - these are in the 8x drive trays
Optical: Pioneer BDR-207M BD-XL drive for long-term archiving
Graphics: Sapphire AMD 7970 with Gelid Icy Vision-A cooler fitted
PSU: Coolermaster Silent Pro 1000W
NIC: Intel E1G42ET Dual Port 1Gbps Server NIC
OS: Currently giving Windows 8 Professional x64 a try, previously Windows 7 Ultimate x64
Not everything currently gets backed up onto my backup machine - for example Steam games don't get backed up here - these go to a separate external drive. I've also replaced the front + rear fans in this with Noctua NF-P12s and added an additional fan in the front.

Backup
Case: Fractal Design Array R2
CPU: i5 2400S @ 2.5 GHz with a Scythe Shurken Rev.B cooler
Motherboard: Intel DH67CF
RAM: 8GB (2x4GB) of Corsair XMS3 DDR3-1600
Non-RAID storage: Crucial m4 128GB - for boot + OS
RAID storage: 4x Hitachi Ultrastar 5K3000 2TB drives in RAID0 (not ideal, but it's for capacity - this is all duplicates of data held elsewhere though) on a 3ware 9650SE-4LPML
OS: Windows Server 2008 R2
Looking at 6x4TB drives for this machine, but the 3ware card will need replacing if I do.


Colo server
CPU: Xeon E3-1230
Motherboard: Supermicro X9SCM-F
Case: Supermicro 113MTQ-563CB
RAM: 16GB (4x4GB) Kingston ValueRAM ECC
Non-RAID storage: Crucial m4 128GB - boot drive - mounted in the internal fixed bay instead of an optical drive
RAID storage: 6x WD5000BPKT 500GB drives in RAID6 on a LSI 9260-8i - mounted in 6 of the 8 2.5" hotswap bays
OS: Xen 4.1 + Gentoo for Dom0, a variety of DomU OSes
This is my personal co-located server - hosts a variety of sites + services. This gets backed up to separate off-site storage via FTP.

That's 27TB of total disk (22TB usable) + a few 2TB USB/eSATA externals which imho don't count.

All of these machines have had surprisingly good reliability - no drive dropouts or RAID rebuilds required since I've set the systems up. Previously I used Western Digital RE GP drives - these ran really hot and they used to constantly overheat and drop out of the array. Pointing some 5000RPM 12cm fans at them solved that but it's crazy loud - which is why I bought the Ultrastars when I needed additional capacity. No regrets there :)
 
SonataSys,

1) Thanks for the detailed explanation

2)

SonataSys said:
The key to RAID0 is purchasing high-quality drives and verifying their surfaces are free of defects early on. In general, a bad disk usually fails early or not at all, meaning disks fail within the first month of use or perform well for many years until service times surpass MTBF.
Click to expand...

What method do you use to achieve the bolded? (Besides naive stress-testing, I cannot think of anything.)
 
Ahriman4891 said:
SonataSys,

1) Thanks for the detailed explanation

2)



What method do you use to achieve the bolded? (Besides naive stress-testing, I cannot think of anything.)
Click to expand...

You are welcome. If you have WD disks you can always download Data LIfeguard, which is a decent surface testing tool:

DataLifeguard.PNG


Whenever I purchase new disks, I immediately use DataLifeguard to perform an extended test, then write zeroes across the entire drive to verify the surface isn't riddled with bad sectors. I have purchased around twenty WD drives so far and I've had to return just one disk, and it wasn't an RE4...
 
SonataSys said:
You are welcome. If you have WD disks you can always download Data LIfeguard, which is a decent surface testing tool:

*snip*

Whenever I purchase new disks, I immediately use DataLifeguard to perform an extended test, then write zeroes across the entire drive to verify the surface isn't riddled with bad sectors. I have purchased around twenty WD drives so far and I've had to return just one disk, and it wasn't an RE4...
Click to expand...

I didn't know you could use WD DLG for that! sweet! I've been looking for something like that! :)

is it possible to use on non-WD HDDs as well?
 
gmutale said:
I didn't know you could use WD DLG for that! sweet! I've been looking for something like that! :)

is it possible to use on non-WD HDDs as well?
Click to expand...

I don't think DLG works with any other brand other than WD, but I've never tried using it on anything but WD, so it might work with other disks. I'm just not sure. There are other, similar tools available that are not designed for a particular manufacturer, although I don't know how useful they are since I've always stuck with WD and its DLG. For example: HDDScan is free and others have told me in the past that it gets the surface test job done well enough...
 
dsc_2738k7jpwz.jpg

Just some storage porn :D Got those for cheap - so I just took them without a big plan. :D
 
I just got some new parts for my server. Deceided instead of running drive bender I needed to run an actual raid card. I have a few areca cards at work, so I wanted to run them at home. I also swapped out my i7 2600 to an i7 3770, got it for free.


Case: Antec 1200
CPU: i7 3770
Mobo: asrock extreme 4 gen 3 z68
RAM: 4x4gb ddr3 1600 corsair vengance
RAID storage: 10x Hitachi deskstar 7k3000 2tb in raid 6. on an areca 1882ix-12 1gb ddr3. I also had 2x2tb seagate 5900 rpm drives that I am running in raid 1. also just thrown in the case, 1 hitachi 7k200 2tb, 1 wd 3tb green, 1 2tb seagate something drive lol. I have 14.6 tb in raid 6.
OS Drive: Corsair force series 3 60gb
PSU: Pc Power and cooling silencer 760 watt
NIC: Intel Gigabit CT desktop
OS: Server 2008 r2

scaled.php
 
Hi, just want to show of the specs of my home server.
It's about 1.5 years old now and have 21 hard drives running.

It has 10x2 TB in zpool2 + 10x3 TB in zpool2, with 250 GB for OS and such.

It is currently running ESXI 5, with Solaris 11 Express and Windows Server 2008 on top of it.
It has 24 GB of DDR3 ECC memory, running a intel W3670 CPU, 3.2 GHz hexacore on a Supermicro X8ST3-F motherboard with IPMI 2.0, I'm running a IBM M1015 with flashed firmware and a HP SAS Expander for SATA distribution.

It ass is packed in a wonderful Supermicro SC846TQ-R1200B case with a lockable front panel.

Things I really want to upgrade but can't afford is 10 GbE, using infiniband or some other cheap solution.

Pictures can be added if wanted :)
 
Rudde93 said:
Hi, just want to show of the specs of my home server.
It's about 1.5 years old now and have 21 hard drives running.

It has 10x2 TB in zpool2 + 10x3 TB in zpool2, with 250 GB for OS and such.

It is currently running ESXI 5, with Solaris 11 Express and Windows Server 2008 on top of it.
It has 24 GB of DDR3 ECC memory, running a intel W3670 CPU, 3.2 GHz hexacore on a Supermicro X8ST3-F motherboard with IPMI 2.0, I'm running a IBM M1015 with flashed firmware and a HP SAS Expander for SATA distribution.

It ass is packed in a wonderful Supermicro SC846TQ-R1200B case with a lockable front panel.

Things I really want to upgrade but can't afford is 10 GbE, using infiniband or some other cheap solution.

Pictures can be added if wanted :)
Click to expand...

two things:

1) pics? YES PLEASE! :D

2) what revision is your mobo? :)

thanks! :)
 
Rudde93 said:
It ass is packed in a wonderful Supermicro SC846TQ-R1200B case with a lockable front panel.
Click to expand...

Big deal putting 21 drives in a case with 24 drive bays... I got 14 drives in Define R3 :D.
fractal-14drives.jpg


8 in the 3.5" bays, 3 in the two 5.25" bays, 3 more in the Lian Li hard drive mount system from PC-A05N. 10 drives are 2TB, 3 are 1TB and there is a 1.5TB system drive, that means 23TB=20.91TiB data storage and 1.36TiB for system & other stuff like virtual machines etc.

Plus the P8B WS, Xeon E3-1235, 24GB RAM, DVB-C TV card and the Intel 2-port mSAS card (ab)used as 8-port SATA HBA.

And yes, it is a mess, try not to have with so much hard drives and so small space :D.
 
Here is my latest build to replace two fully populated HP Micro-servers,

SuperMicro X9SCM-F Board
Intel Xeon E3-1230 v2 3.3GHz
16GB ECC RAM (4x4GB DDR3-1333)
HP Smart Array P410 256MB cache
HP SAS Expander 24 Port
X-Case RM 424 - 24 Hotswap Bay 6GB Mini SAS (SATA/SAS Backplane) - Rail Kit included

Currently just installed VMWare ESXi 5.1 and Windows 2012 Server on two different boot setups to verify hardware is working. Plan is to convert it to an all-in-one. :D

Some pics....
 
Last edited:
I think I have a sickness, more build pictures to come, but I'm now up to three shelves, 100GB of ram, and 70 TB of space.
 
My Home Server

Total Storage: 50.25 TB

Rack Chassie: Supermicro SC846TQ-R1200B, 4U 24-bay Hot-swappable Drive Bays, Redundant PSU, 1200W gold certified (Supermicro MCP-210-84601-0B front panel for physical security.)
Motherboard: Supermicro X8ST3-F Rev 2.0 (BTW, don't use the SAS controller at that board, it fuckings sucks)
CPU: Intel Xeon UP W3670 3.20 GHz Hexacore.
Memory: KINGSTON 24GB 1333MHZ DDR3 ECC CL9 DIMM
SATA Handeling: IBM ServeRAID M1015 flashed to LSI9211-IT mode and a HP SAS Expander (green edition) for distrubution.
Hard drives:

For OS I use SAMSUNG Spinpoint MP4 HM250HJ 250GB 7200 RPM

Storage: ZFS Zpool2 10x Samsung 2TB F4EG (5400 rpm)
ZFS Zpool2 10x Seagate Barracuda ST3000DM001 3TB S-ATA3 (7200 rpm)

The systems runs ESXi 5.0 and I run Solaris 11 Express for filesystem on top of it together with Windows Server 2008.
And sometimes I boot up linux for the fun of it.

No battery backup (yet, any recommendations?)

Future projects are 10GbE, when I can afford it, for now I'm looking into Infinband Ethernet

DJBhp.jpg


QqI2J.jpg
 
Case: Thermaltake Armor (original)
CPU: AMD Athlon II X2 240
Mobo: ECS 885GM-A2
RAM: 4x8gb DDR3 1333 GSkill Ripjaw
RAID storage: 2x240GB Intel 330 SSD Cache mirror, 8x2TB Seagate 7200RPM storage
OS Drive: 2x80GB mirror
PSU: Corsair TX850
NIC: Intel PRO/1000 VT quad port
OS: Windows Server 2012

I've set up the 8x2TB drives in one big Storage Pool and have begun carving up file share and virtual machine storage from it. So far there is a 6TB file share virtual disk in parity mode and two 700GB virtual disks in mirror mode for VMs. I'm using a program called FancyCache to dedicate 4GB of RAM and 40GB of SSD space to each 700GB disk as read/write cache.
 
Have mine finally setup :D (I don't see the point of pictures of mine though, as I am sure everyone knows what a micro server looks like)

HP N40L Micro Server
16GB DDR3 (It Works :D )
Modified BIOS
120GB Intel 510 SSD Boot drive
4x 3TB WD Red HDDs

Running NAS4Free 9.1.0.1 rev 344
Drives setup as ZFS Striped array
Transfer speeds are 70-100MB/s over GbE

Only running CIFS/SMB at the moment, but still need to setup AFP for my various Macs for Time Machine backups.
Will be attempting to setup VirtualBox later on so as to run a Windows VM for the security camera software.
The SSD I thought would be good for running a couple of VMs on.
 
Gotta love those small N40L's, they go for cheap for just a barebone system of around $130-140 online.

What's this modified BIOS you have? (Bigger support for 16gigs of goodness? :D).
 
I guess i can finally join the club!

26 TB Total advertised storage
22 TB Total usable storage

Chassis: SuperMicro SC743TQ-865B-SQ + SuperMicro CSE-M35TQB
PSU: SuperMicro 865W Super Quiet 80 Plus
Motherboard: Asus P8B-C/SAS/4L
CPU: Intel Xeon E3-1240
RAM: 4x Kingston ValueRAM DIMM 4 GB ECC DDR3-1333
GFX: EVGA GeForce GTX 550 ti
Hard Disks: 13 x Western Digital WD20EARX RAID 6 (idle-timer disabled)
SSDs: 1 x Samsung 830 128 GB
OS: Ubuntu 12.04
UPS: APC Smart-UPS 1500

The 22 TB RAID 6 is encrypted using LUKS and on top of that runs a LVM with some logical volumes for user-data, photos, media, backups, vm's and a general share.

The drives and the crypt/lvm setup do a pretty good job. I can read/write at 1.2 GB/s peaks, but 600-700 MB/s is more usual. Only problem is that i think the iowait is a little bit higher compared to before a upgrade (max 20% under heavy IO... added 5 drives and setup up the lvm, before it was a single ext4 partition which i couldn't resize to 22 TB, so i lvm'd it...)

This rig was my home-server for a couple of month, till i decided to turn it into my workstation. So i added a cheap NVidia card because i made some terrible experiences trying to get my HD6850 to run a 24" and a 19" tft.

On the server-side, it runs a Samba server and a MariaDB server for the XBMC-library accross my house. I currently have 3 small XBMC-clients (AMD E350 etc) and 4-8 clients attached, depends on if they are at home or connected via OpenVPN.

I really like the case because it's a real workstation case but i wish the PSU would have a modular cable system, i don't wan't to cut those unnessecary cables.

The only thing i need to change is the position of the SSD. It currently lies on the bottom of the case. Maybe i can attach the drive to the motherboard tray.

The whole system takes about 80 W idle and 160 W at load. With the workstation, 3 monitors (2x 24", 1x 19") and a Windows rig (Intel i7-3770k, 16 GB RAM, Radeon Hd 6850), everything idles at about 280 W. (I need to replace the older TFT's for newer IPS-panels with LED backlight). If everything is fully loaded, 450 W are consumed.

gNMuA.jpg


57wXy.jpg
 
Matthew Kane said:
Gotta love those small N40L's, they go for cheap for just a barebone system of around $130-140 online.

What's this modified BIOS you have? (Bigger support for 16gigs of goodness? :D).
Click to expand...

Sorry, my bad - the bios doesn't help with 16GB - this just seems to be luck with certain brands more than anything else.

The BIOS enables hidden features in the bios including making all ports E-SATA to enable hot swap.

http://homeservershow.com/hp-microserver-n40l-build-and-bios-modification.html
 
Beert3Okad nice setup! I've got something very similar (same chassis and 5-in-3) and running Linux (raid 6/dm-crypt/lvm/ext4)...I hit the same 16TB limit you did for ext4, but I have plans to recreate the filesystem to a 64bit version using unused space and migrating the data over.

Beert3Okad said:
The only thing i need to change is the position of the SSD. It currently lies on the bottom of the case. Maybe i can attach the drive to the motherboard tray.
Click to expand...

I have two SSDs mounted to the empty space near the power supply using a bracket and some velcro. It feels secure enough, it would be nice though to properly mount the bracket using screws.

gQMMp.jpg


trzzb.jpg


1J6h4.jpg


CymQy.jpg


You can see my setup here, if you would like.
 
@BobbyTables
Thanks for the information. Guess i will mount it beside the mainboard with some L-brackets attached to the many free mainboard screwing holes. I will let you know when i've done it. I also want to rework the cables. I don't need the gpio/i2c cables, furthermore i think i desolder some of the unused power-cables and shorten the ones to long. Only i wish the case would hold 3x 120mm fans instead of these 80mm. I mean, i have them running on low speed and they are still a little bit noisy compared to my other computers 120mm fans. But hey... it's "whisper quiet"... at least the power supply..

btw. nice name ;)
 
Thread Crapping is such a strong word. :rolleyes:

@BobbyTables
That case is great looking. I've been searching and searching for something that is not rackmount but has hotswap and I always figured I'd go with 5x3.5" drive modules in a standard case, but that one is perfect. Now just to find one at a reasonable price. :)
 
shaitan667 said:
Have mine finally setup :D (I don't see the point of pictures of mine though, as I am sure everyone knows what a micro server looks like)

HP N40L Micro Server
16GB DDR3 (It Works :D )
Modified BIOS
120GB Intel 510 SSD Boot drive
4x 3TB WD Red HDDs

Running NAS4Free 9.1.0.1 rev 344
Drives setup as ZFS Striped array
Transfer speeds are 70-100MB/s over GbE

Only running CIFS/SMB at the moment, but still need to setup AFP for my various Macs for Time Machine backups.
Will be attempting to setup VirtualBox later on so as to run a Windows VM for the security camera software.
The SSD I thought would be good for running a couple of VMs on.
Click to expand...

Well this is a pic thread so yea I'd say you should post some ;)
 
Sorry for quoting a post from September, but how loud is that Supermicro chassis (SC846TQ-R1200B)?

Rudde93 said:
My Home Server

Total Storage: 50.25 TB

Rack Chassie: Supermicro SC846TQ-R1200B, 4U 24-bay Hot-swappable Drive Bays, Redundant PSU, 1200W gold certified (Supermicro MCP-210-84601-0B front panel for physical security.)
Motherboard: Supermicro X8ST3-F Rev 2.0 (BTW, don't use the SAS controller at that board, it fuckings sucks)
CPU: Intel Xeon UP W3670 3.20 GHz Hexacore.
Memory: KINGSTON 24GB 1333MHZ DDR3 ECC CL9 DIMM
SATA Handeling: IBM ServeRAID M1015 flashed to LSI9211-IT mode and a HP SAS Expander (green edition) for distrubution.
Hard drives:

For OS I use SAMSUNG Spinpoint MP4 HM250HJ 250GB 7200 RPM

Storage: ZFS Zpool2 10x Samsung 2TB F4EG (5400 rpm)
ZFS Zpool2 10x Seagate Barracuda ST3000DM001 3TB S-ATA3 (7200 rpm)

The systems runs ESXi 5.0 and I run Solaris 11 Express for filesystem on top of it together with Windows Server 2008.
And sometimes I boot up linux for the fun of it.

No battery backup (yet, any recommendations?)

Future projects are 10GbE, when I can afford it, for now I'm looking into Infinband Ethernet

DJBhp.jpg


QqI2J.jpg
Click to expand...
 
It has three level you can choose from in BIOS, on the highest performance it pretty damn loud! But in the quiet mode I can have it in a working environment.
 
I'll likely have it in an unfinished basement. I wonder if that would be quiet enough where I won't hear it on the first floor. I originally had a Norco 4224 in that area and the fans were screamers. Definitely heard that on the entire first floor. Unfortunately I've never had good luck with that case. Been through a few backplane replacements, power supplies, and raid controllers and always had issues with drives dropping from arrays.


Rudde93 said:
It has three level you can choose from in BIOS, on the highest performance it pretty damn loud! But in the quiet mode I can have it in a working environment.
Click to expand...
 
Status
Not open for further replies.
Back
Top