The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

[okeygrak]

I've been following this thread for a while, and found some good inspiration for this build.

The goal of this machine was to be near silent and to have very low power consumption.

It is a headless FreeNAS machine, running ZFS as the file system. It's primary function is a media streaming server, and it has been placed directly into my media rack in the living room ( hence the silent requirement ) . I mounted some rubber stoppers inside the SATA racks to minimize vibration ( quite a bit initially ). Overall it is near silent and can only be heard within 2 ft of the unit.

Hardware:
Lian Li A16 Silver Aluminum Case
Gigabyte GA-790XTA-UD4 AM3 Motherboard
AMD 235e 45W Dual Core CPU - Undervolted - Scythe Mugen-2 Heatsink ( passive )
4GB Low Voltage DDR 3 Ram
9 x Samsung Spinpoint Ecogreen F3 2TB Hard Disks
4GB USB Disk on Module from Transcend ( Plugs directly into the USB header pins on board)
Seasonic Gold X650 power supply
Nexus Real Silent 120mm Fans x 2 ( running at stock 1000rpm )

Setup:
ZFS - Raid-Z2 (14TB useable, 4 TB parity )

Performance:
Read / Write ( single client ) : R: 80-85mb/sec | W: 85-95mb/sec

Total System Power Consumption:
Idle ( Disks Spun Down ) : 43 watts
Load ( Disks spun Up ) : 66 watts

Temperatures:
Disks (Idle) : 30 C avg.
Disks (load): 38 C avg.
CPU : 20 idle - 28 C load

 

[croakz]

Hardware:
Lian Li A16 Silver Aluminum Case
Gigabyte GA-790XTA-UD4 AM3 Motherboard
AMD 235e 45W Dual Core CPU - Undervolted - Scythe Mugen-2 Heatsink ( passive )
4GB Low Voltage DDR 3 Ram
9 x Samsung Spinpoint Ecogreen F3 2TB Hard Disks
4GB USB Disk on Module from Transcend ( Plugs directly into the USB header pins on board)
Seasonic Gold X650 power supply
Nexus Real Silent 120mm Fans x 2 ( running at stock 1000rpm )

Nice setup. Do all your drives connect directly to the motherboard?

 

[faugusztin]

You can see it, no ? 8 connected to the right angled SATA connectors, 1 connected via that red cable to the eSATA at back panel.

 

[okeygrak]

You can see it, no ? 8 connected to the right angled SATA connectors, 1 connected via that red cable to the eSATA at back panel.

Correct. This mobo has 8 internal sata, and 2 esata. I am routing 1 esata back inside the case for the 9th drive. It has presented no issues.

 

[98_1LE]

First post, so be nice.

This is my current unRAID server. I am in the process of building another but it isn't done.

MSI 880GMA-E53 mobo
AMD Sempron 140 CPU
Corsair CX430 PSU
Kingston 4x1GB DDR3 1333 RAM
SNT SNT-SAC3141TL 4-in-3 drive cage
old Antec Sonata case
8 Seagate ST32000542AS = 11TB usable after parity & spare
unRAID 4.7 Pro

root@unraid:/mnt/user# df -h /mnt/disk? /mnt/user
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              1.9T  1.8T   79G  96% /mnt/disk1
/dev/md2              1.9T  1.8T   73G  97% /mnt/disk2
/dev/md3              1.9T  1.8T   87G  96% /mnt/disk3
/dev/md4              1.9T  1.8T   76G  96% /mnt/disk4
/dev/md5              1.9T  970G  894G  53% /mnt/disk5
/dev/md6              1.9T   33M  1.9T   1% /mnt/disk6
shfs                   11T  8.0T  3.0T  73% /mnt/user

Write speeds over NFS without parity: 64.9 MB/s (4 TB of BD images)
Write speed over NFS with parity: 22.7 MB/s (2 TB of DVD images)

Full details here


The new box will have 14 external disks and an internal SSD, not counting the boot device (USB flash).

 

[Red Falcon]

I think maybe you are the only one? I know many others who don't get good results on samba as well. You get > 100 megabytes/sec What OS -> What OS?

Only Win Vista and 7 have Samba2, a proprietary file sharing system which can achieve more than the 40MB/s limit with the original Samba with every other OS.

Using Samba, the only way to normally get above 40MB/s is to use a Vista/7 system to another Vista/7 system. In some special scenarios it is possible to get above 40MB/s when not using Vista or 7, but the setups are rare.

Using NFS tends to achieve much better performance and results, though it is designed for a more permanent setup, where as Samba is quick and simple and supported by most OSes.

 

[luca_]

I've been following this thread for a while, and found some good inspiration for this build.

The goal of this machine was to be near silent and to have very low power consumption.

It is a headless FreeNAS machine, running ZFS as the file system. It's primary function is a media streaming server, and it has been placed directly into my media rack in the living room ( hence the silent requirement ) . I mounted some rubber stoppers inside the SATA racks to minimize vibration ( quite a bit initially ). Overall it is near silent and can only be heard within 2 ft of the unit.

Hardware:
Lian Li A16 Silver Aluminum Case
Gigabyte GA-790XTA-UD4 AM3 Motherboard
AMD 235e 45W Dual Core CPU - Undervolted - Scythe Mugen-2 Heatsink ( passive )
4GB Low Voltage DDR 3 Ram
9 x Samsung Spinpoint Ecogreen F3 2TB Hard Disks
4GB USB Disk on Module from Transcend ( Plugs directly into the USB header pins on board)
Seasonic Gold X650 power supply
Nexus Real Silent 120mm Fans x 2 ( running at stock 1000rpm )

Setup:
ZFS - Raid-Z2 (14TB useable, 4 TB parity )

Performance:
Read / Write ( single client ) : R: 80-85mb/sec | W: 85-95mb/sec

Total System Power Consumption:
Idle ( Disks Spun Down ) : 43 watts
Load ( Disks spun Up ) : 66 watts

Temperatures:
Disks (Idle) : 30 C avg.
Disks (load): 38 C avg.
CPU : 20 idle - 28 C load

That is one nice setup! I love the case and the wiring job is nice as well..
I too like to silent my pcs down to the minimum.. good job

Oh, btw, why did you go for the single Max4 HDD Racks instead of 4 in 3 or 5 in 3 backplanes?

 

[okeygrak]

That is one nice setup! I love the case and the wiring job is nice as well..
I too like to silent my pcs down to the minimum.. good job

Oh, btw, why did you go for the single Max4 HDD Racks instead of 4 in 3 or 5 in 3 backplanes?

I went for the single racks because they allow much more spacing between drives, hence making it easier to cool the inside of the case. Those 5 in 3 backplanes cram the drives together, and use a loud 80mm fan behind the drives to move air.

 

[war9200]

Yes, but what speed do you get to the disk over gigabit?
That is cheating, to /dev/null.

 

[luca_]

Yes, but what speed do you get to the disk over gigabit?
That is cheating, to /dev/null.

Not really. If your testing network bandwith it makes more sense to "write" to /dev/null..
Of course this doen't tell you the kind of bandwith you would get with day to day usage.

But saying I can't utilize gigabit because your disk is slower then your network is wrong..
To max out gigabit you would have to have some kind of raid setup or an SSD.

 

[war9200]

Hi,

6 RAPTOR RAID-0: 320MiB/s write on my ATOM
NETWORK -> 320MiB/s RAID-0 = 50-70MIB/s (SAMBA), 70-90MiB/s (FTP/NFS)

Not 120MiB/s like my other hosts.

 

[luca_]

Ok, that is weird.

You should do some googleing, there are a couple of options to maybe improve samba performance (like socket options or write cache).
If you see unstable transfer rates you also might want to check your cables, if they are somehow damaged or not shielded properly this can have an effect as well.
And you could also have a look at CIFS.. but thats getting off topic here

 

[loftydan]

Amount of Raw disk space: 12.81 TB

Case: Cooler Master Elite 330 Mid Tower Case
PSU: Corsair HX750 Power Supply
Motherboard: SuperMicro X8SIL-F-O
CPU: Intel Xeon X3440 Lynnfield 2.53GHz
RAM: 16 GB ECC Kingston ValueRAM / 6 GB assigned to Solaris Express 11
GPU: Onboard
Controller Card:IBM ServeRAID BR10i 8-Port SAS RAID Controller+flashed to IT mode
Optical Drive: LITEON LH-20A1L DVD+RW
Hard Drives: 3 WDC WD20EARS 2 TB , 3 Seagate ST32000542AS 2 TB, 1 Seagate (ST3750330AS) 7200.11 750 GB, 1 60 GB OCZ Vertex 2 SSD (ESXi, Solaris, Win 2008 R2)
Battery Backup Units: None at this time
Operating Systems: ESXi 4.1, Solaris Express 11 with napp-it web GUI, Windows Server 2008 R2, Windows 7, pfSense 2.0

It all started when I just wanted a desktop computer, that could also hookup to the TV. This was a Q6600, Asus P5E-VM HDMI, OCZ StealthXStream 500W with 3 x the 750 GB Seagate drives that I still have one of. Then the drives got updated to 2 TB each. RAID was handled by Intel Matrix RAID.

At some point I found this forum, ZFS, Gea's napp-it interface. So I switched to ESXi, and started testing using Nexenta and a RDM hack. It was dead slow, about 10-20 MB/s, but worked okay for laptops backing up over wireless.

The next upgrade was getting a motherboard, cpu, etc so that I could passthrough the storage controller to a ZFS supporting OS (see above hardware). Getting an LSI 1068e controller into Canada proved to be difficult, until I finally found some on eBay (the IBM card listed above). With help from some members here, I finally that working nicely.

I started to benchmark my system, and the whole thing would restart on high disk activity. Past experience told me the PSU was underpowered. A bit of searching told me my PSU was multi rail, so I replaced it with the single rail Corsair list above.

The system now provides storage for TV shows, movies, music, and Time Machine backups. All photos from our 2 laptops are backed up to the ZFS pool as well. All of the content is then shared using the new Audiogalaxy and PS3 Media Server. It was a long road, but I am finally happy with my home server.

dd write 20 GB @ 247 MBs
dd read 20 GB @ 374 MBs
See this post of the rest of my benchmark results.

Everyone on this forum has been amazing in helping me achieve this result, so thank you. If you have any questions, please ask.

(click on any picture for the full sized image)



My SAS breakout cables are a bit over length for this case, but it all fits okay


Drives numbered 1-6 are for ZFS use


Despite the look, no wires are tangled. When the TV stand is slide forward and angled out, the cables are quite organized.


How it all looks from the front

 

[JensS]

Amount of total storage: 28,32TB

Amount of storage in the following system: 7,5TB

Case: Silverstone FT01
PSU: Enermax Liberty 400W
Motherboard: Asus P7P55D
CPU: i7-860
RAM: 4x 2GB G.Skill RipJaws
GPU: MSI GTX460 Cyclone
Controller Cards: Intel Pro 1000PT DualPort NIC
Optical Drives: LG DVD-RW
Hard Drives: WD1001FALS for sys
Samsung HD502HJ for data
3x Samsung HD204UI for data
Operating System: Win7 Ultimate

Main box for everyday usage, gaming,...
Backup is done to the storage-box as far as I don't forget it

Amount of storage in the following system: 19,32TB

Case: Chieftec CA-01
PSU: Enermax 353W
Motherboard: Asus P5WDG2-WS Pro
CPU: E6750
RAM: 4x 1GB ECC (various)
GPU: Matrox G550 Dual-DVI, to be replaced by some PCI-card
Controller Cards: 2x Dell Perc 5i, HP NC7170 DualPort-NIC
Optical Drives: none
Hard Drives:
Seagate 7200.4 320GB for system
5x Seagte 5900.12 2TB @ RAID5
3x Seagate 7200.11 @ RAID5
3x Samsung HD103SI
3x WD10EADS
2x WD10EACS
Operating System: Win7 Ultimate

My big storage, used as backup-target and archive for files not needed everyday, also to be used as filer at LAN-parties

Amount of storage in the following system: 1496GB

Case: None, built onto particleboard, mounted in a LackRack
PSU: 400W FSP
Motherboard: Asus P5WDG2-WS
CPU: Pentium D 930
RAM: 4GB Nanya
GPU: MSI GeForce 7300LE
Controller Cards: HP SmartArray 6400, HP SmartArray P400
Optical Drives: none
Hard Drives:
2x Seagate Cheetah 15K.4 36GB @RAID1 for Sys
3x Seagate Cheetah 15K.5 73GB @RAID5
4x Seagate 36ES 36GB @RAID0
12x various 18GB/10K (not counted for amount of storage, to be replaced by 10K.7 146GB )
8x Seagate Savvio 10K.2 146GB @RAID0
Battery Backup Units: for both controllers, the packs of the 6400 got fresh cells some days ago
Operating System: W7 or maybe 2k8R2

Just built to have, as the stuff was available for free or so cheap I just had to buy...
Currently it has no real use, maybe I'll use this one as LAN-Party-filer as the big storage-box is damn heavy...

Pics are to follow the next days as I'm a bit lazy right now

 

[cybertron]

Guys, I have a question. I'ev got a chassis with 12 hotswap drives, 8 are sata, and 4 are 15k sas. I have all of this in my bedroom in a rack with 1 other computer and as we leave winter (in atlanta), my room is reaching epic temps, but the moment you walk into my living room, its cold. What are you guys doing to keep the temp down in the room where all your gear is? I'm about to move to a loft that has no ceilings and I'm afraid my entire unit might become massive hot. Can i port the hot air outside or something? There is exposed venting for the dryer and for the bathroom fans, maybe I could tie into one of those and make the machine air tight? ugh, I'm at a loss.

 

[Dasimpson]

if you put it in the loft you could always pipe that heat to other rooms

 

[faugusztin]

Upgraded from Atom 330 board to MSI H55M-ED55 + Core i3 530. When all mechanical hard drives are sleeping the system uses 41W from socket, when none of them is in standby it is 73W.

Sorry for the dust and duct tape instalation of SSD .

Exact specs :
Core i3 530 cooled by Coolink Corator DS (without fan)
MSI H55M-ED55
2x2GB Corsair Dominator GT (i didn't want to unbox the unused XMS3 modules i got back from RMA )
Intel SASUC8I controller
6xWD20EARS
4xWD10EADS
Seasonic X-650
3x Noiseblocker Multiframe fans

 

[J-Will]

Upgraded from Atom 330 board to MSI H55M-ED55 + Core i3 530. When all mechanical hard drives are sleeping the system uses 41W from socket, when none of them is in standby it is 73W.

Sorry for the dust and duct tape instalation of SSD .

Exact specs :
Core i3 530 cooled by Coolink Corator DS (without fan)
MSI H55M-ED55
2x2GB Corsair Dominator GT (i didn't want to unbox the unused XMS3 modules i got back from RMA )
Intel SASUC8I controller
6xWD20EARS
4xWD10EADS
Seasonic X-650
3x Noiseblocker Multiframe fans

sounds like a nice setup, and clean. What OS are you using?

 

[faugusztin]

Arch Linux. Tried running mprime, it tops at 70-71C on one core and 65C on second, while running 4-thread test for half a hour. Considering it runs semi-passive, i think the results are good enough.

Because it is just a file storage, i don't use any kind of RAID setup => 16TB claimed space, 15.25TiB if i count correctly.

 

[longblock454]

Updated list will be posted soon.

Please remember to PM me with your post #, total storage, most in a single chassis and OS stats.

 

[a little off]

Amount of total storage: 22.56 TB

Desktop = 6.56TB

4x WD SE16 Blue's 640GB
4x Hitachi Deskstar 1TB



Server = 16TB

4x WD20EARS 2TB
4x Samsung HD204UI 2TB

Grover:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              52G  1.1G   48G   3% /
none                  1.6G  268K  1.6G   1% /dev
none                  1.6G     0  1.6G   0% /dev/shm
none                  1.6G  752K  1.6G   1% /var/run
none                  1.6G     0  1.6G   0% /var/lock
/dev/md1              5.4T  187M  5.1T   1% /storage01
/dev/md0              5.4T  1.2T  4.0T  22% /storage00

 

[Butcher9_9]

any reason you use so little of your storage?

 

[a little off]

any reason you use so little of your storage?

I have a ton of space on the desktop because I got the 4 Hitachi's for $150...

The server is kind of an emergency rebuild and I am in the process of moving everything back too it. When it's all transferred back, it will sit at about 30-35%

 

[ArnisLav]

Exact specs :
Core i3 530 cooled by Coolink Corator DS (without fan)
MSI H55M-ED55
2x2GB Corsair Dominator GT (i didn't want to unbox the unused XMS3 modules i got back from RMA )
Intel SASUC8I controller
6xWD20EARS
4xWD10EADS
Seasonic X-650
3x Noiseblocker Multiframe fans

What case are you using? Looks interesting to me. 8X3.5" HDD bays and still not big one.

 

[faugusztin]

That was a Fractal Design Define R2, not sold anymore. They are selling R3 in Europe now (small changes to the better), in USA you can get them only at ++++ or sometime soon at Newegg.

 

[dodgeboy]

NAS1: ESXi / OpenSolaris 11 / ZFS
37TB online (48TB advertised)

Case: Norco 4224
PSU: Antec 1200 Quattro
Motherboard: Supermicro X8DT3-LN4F
CPU: 2x Intel E5606 Xeons
RAM: 48GB (12x 4GB Crucial ECC)
Controller: 3x LSI 1068E (onboard and 2 cards, using PCI Passthrough in ESXi)
Expander: HP SAS Expander (currently not in use)
Boot drive: Crucial 128GB SSD
Hard drives: 24x Hitachi 5k3000 (HDS5C302)
Network: 4 on-board Intel gig NICs, etherchanneled
Operating System: OpenSolaris 11 (running under ESXi 4.1, on the same host); allocated 24GB of RAM and 4 cores
File System: ZFS RaidZ2 + 1 hot spare & 120GB SSD for ZIL and L2ARC caching, with compression enabled
Battery Backup: APC Smart-Ups Net 3000VA

Filesystem             size   used  avail capacity  Mounted on
raid/share              37T   668G    37T     2%    /raid/share

NAS2: Ubuntu Hardware RAID
24TB online (30TB advertised)

Case: Norco 4020
PSU: Antec 900
Motherboard: Asus
CPU: Intel Core i5 720
RAM: 4GB (2x 2GB Crucial)
Controller: Areca ARC-1280ML-2G
Boot drive: Seagate 750GB
Hard drives: 20x Seagate 2TB ST31500341AS
Network: 2 Intel gig NICs, etherchanneled
Operating System: Ubuntu 9.10
File System: XFS
Battery Backup: APC Smart-Ups Net 1400VA

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1              24T   20T  3.6T  85% /media/raid

Please PM me if you have questions. This thread is too active to monitor.

 

[bluemonkeyfish]

as i look around the forum, i am excited to take my current array (10x750GB [2 parity]) and migrate to 16x2TB (2 Parity) with the following...


1x AREC-1880x-8
1x Areca ARC-8026-16
16x WD RE4 2TB WD2003FYYS

and then upgrade the 2nd server to 16x2Tb (2 Parity)

1x Areca ARC-8026-16
16x WD RE4 2TB WD2003FYYS

with an iscsi target on a windows 2008 storage server.

(My Lab/Network)

http://www.flickr.com/photos/daniel_messana/

 

[OldSchool]

wow nice rack there bluemonkeyfish

 

[98_1LE]

Basic specs: MSI 880GMA-E53 | AMD Sempron 140 | Seasonic SS-400FL | Kingston 4x1GB DDR3 1333 | Icy Docks| Antex 902v2 case | 8 ST32000542AS + 4 WD20EARS with one parity and one spare | internal 32GB SSD | unRAID 4.7 Pro

The internal SSD is for read cache of xml & jpg for media players.

I tried to make the cables neat, but there are too many cables and too few options on length.

A thermal switch activates the 200mm and 120mm case fans. The rest of the time the only fans running are the ones in the Icy Docks, which set on low are very quiet; plus a slow/quiet 120mm fan in the side of the case blowing on the HBA, north bridge and CPU cooler. I used a hair dryer to test the switch. It never gets hot enough, even with a passive CPU cooler. There is room to add two more disks but I have enough space for a good long time.

This df was taken as I typed this post.

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             3.8G  133M  3.6G   4% /boot
/dev/md1              1.9T  1.8T   67G  97% /mnt/disk1
/dev/md2              1.9T  1.8T   71G  97% /mnt/disk2
/dev/md3              1.9T  1.8T   66G  97% /mnt/disk3
/dev/md4              1.9T  1.8T   68G  97% /mnt/disk4
/dev/md5              1.9T  1.8T   70G  97% /mnt/disk5
/dev/md6              1.9T  522G  1.4T  28% /mnt/disk6
/dev/md7              1.9T   33M  1.9T   1% /mnt/disk7
/dev/md8               30G   60M   30G   1% /mnt/disk8
/dev/md9              1.9T   33M  1.9T   1% /mnt/disk9
/dev/md10             1.9T   33M  1.9T   1% /mnt/disk10
/dev/md11             1.9T   33M  1.9T   1% /mnt/disk11
shfs                   19T  9.3T  9.0T  51% /mnt/user

A very detailed description of the build can be found here is anyone cares.

Comments or suggestions appreciated. I know the high speed ZFS systems are the rage here but for what I need this is working out great.

 

[Cryptic1911]

Here's my setup.. Just added 5 2TB drives last night (haven't moved anything to them yet), so now I'm in the club! lol

Norco RPC-4020 4U Rackmount case w/ 20 drive bays - modded with 3x 120mm fans in backplane instead of 5x80mm
Corsair 750W power supply
Gigabyte GA-N680SLI-DQ6 motherboard (4x gigabit nics, 3x 16x PCIe slots)
Intel Q6600 Core2Quad 2.4ghz
8GB Corsair XMS2 DDR2 memory (mobo is old and extremely picky w/ 4 sticks of ram- had to run at 711mhz or it wouldnt boot)
ATI Radeon 9250? 128mb PCI videocard (fanless - smallest card I had lol. Will probably pull since its setup now)
2x Intel SASUC8I 8 port Sata HBA's flashed with LSI 3801e IT firmware
4x LSI SFF8087 -> 4x Sata forward breakout cables ( to backplanes on case)
2x 80 Gb Sata disks for boot (one WD , one Seagate)
10x Seagate 7200.10 500gb 16mb cache Sata disks (getting a bit old, but they still work fine)
5x Samsung HD204UI 2TB 32mb cache Sata disks aligned to 4k sectors (just got these in last night, so still empty - was testing speeds and making sure I got the 4k alignment right)

Running FreeBSD 8.2 AMD64 w/ RootOnZFS install
zfsroot is a pool with a 2 disk mirrored vdev running on a WD 80gb Sata disk, and a Seagate 80gb Sata disk
zfsarray is a pool with two 5 disk raidz vdevs w/ the Seagate drives
zfsarray2 is a pool with one 5 disk raidz vdev w/ the Samsung drives

Serving:
ZFSGURU <- installed the 0.1.8 script only version on top of BSD. User sub.mesa on the forum is the developer - http://www.zfsguru.com
SabNZBD <- downloads my NZB files for movies and such through USENET http://www.sabnzbd.org
Sickbeard <- searches for TV shows on NZB sites, then hands NZB's of to SABnzbd, then this gathers nfo and thumbnails for shows plus sorts them http://www.sickbeard.com
Couchpotato <- like Sickbeard, but does movies instead http://www.couchpotatoapp.com
Webmin
sshd
Samba
Dnsmasq
DHCP
and I'll be setting it up as the firewall / router once I stop being lazy

zfs tuning params:

vm.kmem_size="12g"
vfs.zfs.arc_min="1g"
vfs.zfs.arc_max="11g"
vfs.zfs.prefetch_disable="0" (don't really need this, just leftover from when I forced it on when I only had 4gb ram)
vfs.zfs.txg.synctime="1"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.min_pending="1"
vfs.zfs.vdev.max_pending="1"
vfs.read_max="128"
vfs.zfs.zil_disable="1" <- I had turned this off initially to do some testing when I had speed issues. don't think it made a difference speed wise, but I'll turn it back on and re-benchmark since its safer to not disable

 

[corge]

Total single system storage: 32TB

Storage Repository for XenServer ISO images, XenDesktop profiles, etc...

Case: SuperMicro SC836TQ-R800B
PSU: 800w redundant
Mobo: SuperMicro X8STE
Proc: Intel i7-950 3.06Ghz
Ram: 6GB G.Skill DDR3
Raid: 2x Areca ARC-1222
HDD: 16x Hitachi 2TB
OS: FreeBSD (ZFS)

Will be connected to the network via 2x Intel Gigabit NICs via LACP.

I will upload the pictures soon. I forgot my usb cable for my iphone.

 

[Yonzie]

Amount of total storage: 29.75TB (25.25TB usable) (1k = 1000 bytes)

Amount of storage in the following system: 7.25TB (7.25TB usable)
Mac Pro (Early 2008)
CPU: Dual Quad core Xeon @ 2,8Ghz
RAM: 16GB (8x2GB)
GPU: Radeon 3870 + nVidia 8800GT
Hard Drive: Crucial RealSSD C300, 256GB
Hard Drive: 2x WD GP 2TB
Hard Drive: WD GP 3TB
Optical Drives: Dual superdrives
Battery Backup Unit: APC Smart-UPS 2200
Operating System: OS X 10.6
Used for watching my porn.

Amount of storage in the following system: 10.25TB (8TB usable)
Case: Antec Threehundred
PSU: Corsair 400W
Motherboard: Asus M4A785TD-V EVO
CPU: AMD Athlon II x2 235e
RAM: 4GB
GPU: onboard
Network: Intel e1000
RAID Controller: LSI SAS3081E-R (JBOD, actually ordered Intel SASUC8I, got this instead.)
Hard Drive: Seagate ST3250824AS 250GB
Hard Drive: 5x Hitachi DeskStar 7K2000 2TB
Optical Drive: Generic DVD drive.
Battery Backup Unit: APC Back-UPS CS 325
Operating System: OpenSolaris
Used for storing my porn. Using ZFS with RAID-Z of course.

Amount of storage in the following system: 12.25TB (10TB usable)
Case: Antec Threehundred
PSU: CoolerMaster 620W
Motherboard: Asus M3A-H HDMI
CPU: AMD Athlon 64 x2 7750 (passive cooled)
RAM: 2GB
GPU: onboard
Network: Intel e1000
RAID Controller: LSI SAS3081E-R (JBOD)
Hard Drive: WD WDC2500AAKS 250GB
Hard Drive: 6x Hitachi DeskStar 7K2000 2TB
Optical Drive: Generic DVD drive.
Battery Backup Unit: APC Back-UPS CS 325
Operating System: OpenSolaris
Used for backing up my porn. Using ZFS with RAID-Z of course.

Ran-chan:~ yonzie$ df -h
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2   238Gi  211Gi   27Gi    89%    /
/dev/disk1s2   1.8Ti  870Gi  992Gi    47%    /Volumes/Music
/dev/disk3s2   1.8Ti  309Gi  1.5Ti    17%    /Volumes/Stuff
/dev/disk2s2   2.7Ti  1.7Ti  1.0Ti    64%    /Volumes/TM

yonzie@Ui:~$ df -h |grep tank
tank                  7.2T  5.2T  2.1T  72% /tank

yonzie@Azusa:~$ df -h | grep tank
tank                  8.9T  4.9T  4.1T  55% /tank

 

[luca_]

That is a lot of money for some porn ..

How do you backup stuff from one host to the other? rsync or zfs send / receive or ... ?

 

[Yonzie]

At the moment I use rsync. Old and proven and I know how to use it, which I can't say about zfs send/receive. It's severely CPU limited, however, so I may look into other options when time allows.

 

[PharetH]

Amount of total storage: 124.8TB - All my numbers are actual formatted size.

Main File Server - Two Norco cases built into one.
Amount of storage in the following system: 90.8TB
Case: NORCO RPC-4220 (Top Case), NORCO RPC-4224 (Bottom Case) + Rackable SE3016
PSU: Corsair 950w (Top Case), Corsair 850w (Bottom Case)
CPU: Intel Quad Core Xeon X3330
Motherboard: Asus P5BV-M
RAM: 8GB (4x2GB)
SSD: Western Digital 64GB SSD
RAID 1: 20x Hitachi 7K2000 HDS722020ALA330 (Raid 6 + Cold Spare) (32.7TB) (Norco RPC-4220)
RAID 2: 16x Hitachi 5K3000 HDS5C3020ALA632 (Raid 6 + Cold Spare) (25.4TB) (Rackable SE3016)
RAID 3: 14x Toshiba DT01ACA300 (Raid 6 + Cold Spare) (32.7TB) (Norco RPC-4224)
Raid Card: Areca ARC-1680IX-24-2G w/BBU + HP SAS Expander
Operating System: Windows Server 2008 R2

Vacation House Media Server/XMBC Machine
Amount of storage in the following system: 34TB
Case: Lian-Li PC-V2000A + Rackable SE3016
PSU: PC Power & Cooling 1000w
Motherboard: Asus P5Q-E
CPU: Intel Q8200
RAM: 4GB (4x1GB)
GPU: MSI 9500GT 1GB
RAID Controller: Promise EX8350
RAID Controller 2: Adaptec 5405 (Runs the Rackable enclosure)
Hard Drive: WD Raptor 150GB
Hard Drive: 8x Random 1.5TB (Raid 0) ~10.8TB
Hard Drive: 1x Samsung 1.5TB
Hard Drive: 1x Western Digital 2TB
Hard Drive: 4x Random 1TB drives Raid 0
Hard Drive: Rackable Enclosure (6x 3TB)
Operating System: Windows 7 Enterprise x64

 

[DieTa]

VERY nice @PharetH

I admit, I'm jealous of all the Norco 19" 4U+ cases used in this thread. I've spent some hours yesterday and re-read the whole thread - it's simply fascinating.

@Ockie: What's your current state?

@houkouonchi: Would love to read (and see - photos maybe? ) of your current state. Have you crossed the 200TB barreer meanwhile?

My plan for today is a complete overhauling of my little network. Somewhere in the past, I lost the overview, so I will repatch it all from a Cat5e to a Cat6 patch (only one, not two as now) and separate network and server(s), also - will try to put my cube into my 15U rack. It simply would look better that way. Of course, I will take some photos and present here (if you guys want to see them).

Oh btw. also changed some stuff on my server last week. Had some bad issues with the Nexenta Core 3 (http://pastebin.com/xhjfXQRL), so I switched to OpenIndiana with @_gea's napp-it and recreated all the stuff from scratch. But more details later

 

[houkouonchi]

@houkouonchi: Would love to read (and see - photos maybe? ) of your current state. Have you crossed the 200TB barreer meanwhile?

It pretty much looks the same as it used to. I haven't been doing any big purchases/drive upgrades since I bought a house. I also still have some breathing room on my main system:

root@dekabutsu: 07:51 PM :~# df -H /data
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sdd1               36T    25T    12T  68% /data
root@dekabutsu: 07:51 PM :~#

About the only change is the fact I have added a 10mbit cable internet connection for backup.

 

[DieTa]

It pretty much looks the same as it used to. I haven't been doing any big purchases/drive upgrades since I bought a house. I also still have some breathing room on my main system:

root@dekabutsu: 07:51 PM :~# df -H /data
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sdd1               36T    25T    12T  68% /data
root@dekabutsu: 07:51 PM :~#

About the only change is the fact I have added a 10mbit cable internet connection for backup.

12TB of free storage is indeed a good space for "breathing"

Here are my updates from the complete Saturday.


Amount of total storage: 23.5TB

Fileserver Karinka (&#12363;&#12426;&#12435;&#12363
Case: Yeong Yang Server Cube YY-B0221
PSU: Some old, but working fine 450W CoolerMaster (next one will be a Corsair)
CPU: AMD 254e 2.9GHz
Motherboard: GigaByte GA-MA770T-UD3
RAM: 16GB ECC (4x4GB Kingston)
Hard Drive System: Western Digital 160GB IDE
Hard Drive Cache: Intel 80GB SSD
Hard Drives "Storage": 6x Hitachi Hitachi HDS721010CLA332 1TB in RAIDz1
Hard Drives "Mediapool": 3x WD 1.5TB + 2x Samsung 1.5TB in RAIDz1
Hard Drive "Backup-Data": 2TB external Hitachi USB2
Hard Drive "Backup-Mediapool": 4x WD 1TB in HW RAID5 (Onnto DataTale via USB2)
Operating System: OpenIndiana w/ napp-it

Usable storage: 5TB "Storage" + 6TB "Mediapool" = 11TB
Usable backup: 2TB "Backup-Data" + 3TB "Backup-Mediapool" = 5TB w/ compression on

dieta@karinka:~# zfs list
NAME                                         USED  AVAIL  REFER  MOUNTPOINT
backup-data                                  483G  1,31T   483G  /backup-data
backup-mediapool                             351G  2,33T   351G  /backup-mediapool
mediapool                                   2,47T  2,89T  49,5K  /mediapool
mediapool/mediapool                         2,47T  2,89T  2,47T  /mediapool/mediapool
[...]
storage                                      490G  3,96T  59,8K  /storage
storage/Backups                             61,7G  3,96T  61,7G  /storage/Backups
storage/Bilder                               295G  3,96T   295G  /storage/Bilder
storage/Dad                                  128M  3,96T   128M  /storage/Dad
storage/Dennis                              5,04G  3,96T  5,04G  /storage/Dennis
storage/Firmendaten                         19,5G  3,96T  19,5G  /storage/Firmendaten
storage/Software                             109G  3,96T   109G  /storage/Software

Yesterday, I spent hours and hours on remodeling (spring-cleaning) my network + the room in the basement. It was way to chaotic in this room and I in the end, I had no clue which port belongs to which device and socket in the house.

How it looked before

For the fileserver, I bought those rubber mat, so it does not sit metal on metal inside the rack. The main idea was to put the server into the rack, because there's (currently) more than enough space inside. I also bought this great cable-test device - not cheap - but worth it. Indeed found a not well patch socket in one room.

I couldn't resist - needed more backup storage (I know about snapshots, but don't trust it 100%).

Removed stuff from the top of this cute rack..

Even inside the small rack, it was a bit messy

Removed all stuff from inside - looks much nicer!

Had to reconnect the phone line first - used the internal socket of the NTBA (ISDN)

Legend:
DB = Dachboden = Attic
AZ = Arbeitszimmer = Office
WZ = Wohnzimmer = Living Room
SZ = Schlafzimmer = Dorm Room
DBTEL = Dachboden Telefon = Fon at the attic (ISDN)
SZTEL = Schlafzimmer Telefon = Fon at the Dorm Room (ISDN)

First lines patched. Forgot that it's annoying to run up & down several times, for every socket and port But now, all sockets/ports are 100% working fine!

The switch is a Cisco/Linksys 16 Port GBit switch - amazing how loud the fan is, gets enough cool air inside the small rack. Used my label printer a lot!

SOME (not all!) of the cables I got out of the racks.

The rubber mat cut and laid into the rack, need 36cm at least, so the IPCop has to move up 1U.

Had an unused VESA mounting kit laying around - why not put it to the wall?

Small rack for network only stuff done. You can see both lines (1x Telekom DSL with 16M/1M and 1x Vodafone DSL with 6M/640k) on the top. The 3 boxes inside the rack are: FritzBox for Vodafone DSL, FritzBox as VoIP router for my Cisco phone in the office and a ADSL2+ modem).

Everything in place!

And working

Having a look at the temps here. 20-22°C is normal when the systems are used.

Maybe it's not perfect, but MUCH better and MUCH EASIER reachable now.

 

[Iron67]

This is my workstation

TOTAL STORAGE: 11 TB (6.36 usable)

Case: Cooler Master ATC-201
PSU: Enermax Liberty ELT500AWT
Mainboard: Asus M2N32-SLI Dlx/WF
CPU: AMD Athlon64 X2 3800+
RAM: Corsair XMS2 Twin2x2048-6400C4
VGA: Asus EAX1950XTX
RAID Card: Adaptec RAID 5805
HDD: 2x Seagate 7200.11 ST31000340AS, 3x WD Caviar Black WD1001FALS, 3x WD RE4 WD2003FYYS
ODD: Plextor PX-W4012TA, LiteON iHAS 120
DTV: Hauppauge HVR-2200

 

[jmroberts70]

What a fantastic basement setup!! Great job!!