[H]ard Forum Storage Showoff Thread

Had to upgrade my rig. For fun I guess. :)

Mobo: SuperMicro X11-SSH-F
CPU: Intel Xeon E3-1240v5
RAM: 2x16GB Samsung DDR4 ECC
Case: Fractal Design Define R5
And a btter PSU

Next: Some more HDDs and migration from BTRFS to ZFS.

What drives do you currently have in the system?
 
Made some upgrades over the last few week. Got 120TB in primary server and 120TB in backup server.

Primary is the top 2 chassis in the below stack. Just got a 24 bay SuperMicro 216 2U chassis that is connected the the main Areca controller in the 846 chassis below it. Starting to fill it with Samsung 840 Pros for "scratch" space.

SuperMicroStack1.JPG


The lower 3 chassis (826, 846, 846) is the backup server.

Primary storage setup as follows:

BramaAreca1.JPG


BramaAreca2.JPG


Secondary storage setup as follows:

Argon%20Array.PNG


I need to upgrade the LSI 9266 controller to something with more SFF-8087 ports so I can get rid of the bottleneck caused by cascading 48 of the 2TB drives over a single 4 channel SFF-8087 link. I'll probably pick up another Areca 1882 like what I have in the primary server.

Backside of the SuperMicro stack. Primary server has 10GigE to the core switch and there is a direct server to server 10GigE link as well.

SuperMicroStack2.JPG


A shot of the switches and dual GigE Fiber ONTs:

Network%20Stack.JPG
 
Made some upgrades over the last few week. Got 120TB in primary server and 120TB in backup server.

Primary is the top 2 chassis in the below stack. Just got a 24 bay SuperMicro 216 2U chassis that is connected the the main Areca controller in the 846 chassis below it. Starting to fill it with Samsung 840 Pros for "scratch" space.

The lower 3 chassis (826, 846, 846) is the backup server.

Primary storage setup as follows:

Secondary storage setup as follows:

I need to upgrade the LSI 9266 controller to something with more SFF-8087 ports so I can get rid of the bottleneck caused by cascading 48 of the 2TB drives over a single 4 channel SFF-8087 link. I'll probably pick up another Areca 1882 like what I have in the primary server.

Backside of the SuperMicro stack. Primary server has 10GigE to the core switch and there is a direct server to server 10GigE link as well.

SuperMicroStack2.JPG


A shot of the switches and dual GigE Fiber ONTs:

Network%20Stack.JPG


I hope this isn't too bothersome for you but I am building my own array and am looking to learn more about the networking side of it. I noticed your network switches and all the cables connected to your servers and am hoping you would be willing to explain your particular setup and how someone like me can duplicate your setup and transfer speeds. I have 4 storage servers that will make up my array.
 
I hope this isn't too bothersome for you but I am building my own array and am looking to learn more about the networking side of it. I noticed your network switches and all the cables connected to your servers and am hoping you would be willing to explain your particular setup and how someone like me can duplicate your setup and transfer speeds. I have 4 storage servers that will make up my array.

The bottom two 4U server chassis just hold disks, a power supply, and an SAS expander if I had to guess. I think the actual server is the 2U chassis above them; you can see the I/O plate and motherboard connectors.

His setup is likely something like this.
Server with HBA/RAID card -> Internal SAS 8087 cable -> 8087 to 8088 adaptor -> External SAS 8088 cable -> 8087 to 8088 adaptor (in the first DAS/disk shelf/expander chassis whatever you call it) -> 8087 cable -> to expander card going to disks and out to the next chassis...

You can buy chassis like his premade or build your own. A cheap and popular option is the Rackables se3016 several are shown in this thread.

Now if you wanted to make some kind of storage cluster, that would be a different beast all together.
 
Had to upgrade my rig. For fun I guess. :)

Mobo: SuperMicro X11-SSH-F
CPU: Intel Xeon E3-1240v5
RAM: 2x16GB Samsung DDR4 ECC
Case: Fractal Design Define R5
And a btter PSU

Next: Some more HDDs and migration from BTRFS to ZFS.


Hi thinking about building something like that
especially with the Fractal design case. How is the temperture on the disks?
 
I am a total nub to all of this stuff but everybody has to start somewhere right? I just bought a Rackable 3016 off Ebay for $130 and planning to run a server off my HP Z800 to access the drives. Quick question about hard drives from you guys who are seriously awesome at this stuff. I have been snagging cheap Hitachi Ultrastar 7k3000 SAS (2 TB and 3 TB) drives off eBay. Most of the time they are direct Hitachi Refurbs in original sealed packaging and ranging from $35-60 each. So far I have 6 x 2TB and 5 x 3TB.

I haven't seen very many people using Hitachi drives and when they do, they are normally deskstars. Am I going down a path I don't want to be down? Either with the SAS drives or with Ultrastars in general?
 
I have 12 3TB drives, split between seagates and Ultrastars. Been using them for the last 2-3 years without a single drive failure. The $70 3TB refurbs have served me pretty well so far. I'd highly recommend them as a cheap way to expand storage. Just make sure to have both redundancy and backups, so that *when* a drive or three fails, you still have all of your data.
 
I have over 300 ultrastar's in service, Various models and sizes. I have been very happy with the performance and failure rate so far. I have them in several se3016 enclosures that run 24/7/365 as well as several synology NAS units. All of them stay at good temps and have good power.

I have only suffered 3 failures so far in 3.5yrs. All 3 have been different batches and different models. All 3 failures were also healed by hot-spares with no issues.

Post failure inspection yeilded an unacceptable amount of read errors caused by bad sectors. The usual bad-hdd result IMHO.
 
Currently I'm just using SAS drives rather than SATA drives in my server. The costs aren't that much more for the 2tb to the 4tb. The 5tb and up SAS drives are crazy expensive compared to the SATA though.
 
Nothing crazy. 9TB FreeNAS server I user for home. Running strong for 2 years now, no drive failures.
 

Attachments

  • NAS Side.jpg
    NAS Side.jpg
    186.8 KB · Views: 397
Thanks for info about Ultrastars. I didnt know about them and now I see they are enterprise drives. :D
 
Had to upgrade my rig. For fun I guess. :)

Mobo: SuperMicro X11-SSH-F
CPU: Intel Xeon E3-1240v5
RAM: 2x16GB Samsung DDR4 ECC
Case: Fractal Design Define R5
And a btter PSU

Next: Some more HDDs and migration from BTRFS to ZFS.

What's prompting the migration from BTRFS?
 
My media server (update 3)

TOTAL STORAGE: 52 TB (34.7 usable)

Chassis: Supermicro CSE-846XE26-R1K28B
Mainboard: Asus M2N32-SLI Dlx/WF
CPU: AMD Athlon64 X2 6000+
RAM: 2x Samsung PC2-6400 2GB
VGA: Asus EAX1950XTX
HBA: Adaptec RAID 5805ZQ
Cache: Samsung SSD 850 Pro MZ-7KE256BW
HDD: 2x Seagate 7200.10 ST3250620AS
HDD: 2x Seagate 7200.11 ST31000340AS
HDD: 2x Western Digital Caviar Black WD1001FALS
HDD: 8x Western Digital RE4 WD2003FYYS
HDD: 8x Western Digital Re WD4000FYYZ
ODD: LiteON iHAS 120
DTV: Hauppauge HVR-2200
FAN: Original PWM fans with thermal probe & two channel controller
OS: Windows 7 Ultimate

DLQ1WYz.jpg


RVmkmQP.jpg


iG02cUZ.jpg


auODkRg.jpg
 
Last edited:
My media server (update 3)

TOTAL STORAGE: 52 TB (34.7 usable)

Chassis: Supermicro CSE-846XE26-R1K28B
Mainboard: Asus M2N32-SLI Dlx/WF
CPU: AMD Athlon64 X2 6000+
RAM: 2x Samsung PC2-6400 2GB
VGA: Asus EAX1950XTX
HBA: Adaptec RAID 5805ZQ
Cache: Samsung SSD 850 Pro MZ-7KE256BW
HDD: 2x Seagate 7200.10 ST3250620AS
HDD: 2x Seagate 7200.11 ST31000340AS
HDD: 2x Western Digital Caviar Black WD1001FALS
HDD: 8x Western Digital RE4 WD2003FYYS
HDD: 8x Western Digital Re WD4000FYYZ
ODD: LiteON iHAS 120
DTV: Hauppauge HVR-2200
FAN: Original PWM fans with thermal probe & two channel controller
OS: Windows 7 Ultimate

Which kind of SAS Expander you are using ?
I know LSI SAS... but which one...

Nice setup by the way
 
They're using the LSI 6gbit one that's integrated into the backplane of the chassis. Chip on it should be the SAS2x36.
 
My media server (update 4)

TOTAL STORAGE: 52 TB (34.7 usable)

Chassis: Supermicro CSE-846XE26-R1K28B
Mainboard: Asus Crosshair V Formula Z
CPU: AMD FX-8370
RAM: 2x Crucial Ballistix Elite DDR3-1866-8GB
VGA: Asus EAX1950XTX
HBA: Adaptec RAID 5805ZQ
Cache: Samsung SSD 850 Pro MZ-7KE256BW
HDD: 2x Seagate 7200.10 ST3250620AS
HDD: 2x Seagate 7200.11 ST31000340AS
HDD: 2x Western Digital Caviar Black WD1001FALS
HDD: 8x Western Digital RE4 WD2003FYYS
HDD: 8x Western Digital Re WD4000FYYZ
ODD: LiteON iHAS 120
DTV: Hauppauge HVR-2200
FAN: Original PWM fans with thermal probe & two channel controller
OS: Windows 7 Ultimate

ulFeHqp.jpg
 
Included links for reference. Not paid to endorse, advertise, or promote any of the products linked.

Total Advertised Capacity: 52TB
Total RAID6 Capacity: 44TB
Current Setup: 20TB RAD6, 16TB RAID6 (see shenanigans* below)

Case: Cooler Master Storm Trooper
HDD Expander: FatCage 5-in-3
PSU: eVGA SuperNova G2 1300w
Motherboard: ASUS P8P67 Deluxe B3
CPU: Intel i7 2600k @ 4.4GHz
RAM: G.Skill Sniper DDR3-2400 4x8GB
Controller Cards (if any): LSI MegaRAID 9260-8i w/BBU and Intel RAID Expander
Hard Drives (include full model number): HGST NAS 4TB, 13x - H3IKNAS40003272SN (0S03664)
Operating System: Windows 10 Pro x64 on Samsung 850 EVO 1TB

I am nowhere near full--the original 5x4TB RAID6 gave way to 7x, then I saw the drives mentioned above on sale (lowest price they've been in years and don't go below that according to the trends I could find). Decided to max out.

Note that if you want to stick a FatCage in the Storm Trooper you have to bend or dremel. From the photos I saw it didn't look like it had tabs in the 5.25" drive bays but it does.

* I have to do some shenanigans. When I was new to RAID (level 5, PERC5i, 3x2TB drives) I figured 4kB cluster sizes were fine and dandy. Except when I went to move from 5x to 7x 4TB drives, things weren't fine and I ended up with a two part dynamic disk since I hit a filesystem limitation.

Now I'm copying everything off the 20TB RAID6 onto the 16TB RAID6. Then I'm going to do an online capacity expansion to get to the full 44TB RAID6. I don't project needing drives again unless a unit dies outside warranty.

All but some replaceable files are on Backblaze.

This is/was my "everything" machine: games, storage, media, work (from home), HTPC. It also weighs over 100 pounds due in large part to the drives (the empty case is 31.7 pounds by itself). I am literally going to a hardware store to buy a hand cart next time I need to move it. LANs will use a "battle box" from now on to save me the hassle.

38ekLhX.png

MegaRAID Storage Manager

zTM8fc8.jpg

Case Front (note the FatCage)

GdssmVR.jpg

Case Side (due to design of case, the drives themselves cannot be seen)
 
IMG_1195.jpg
IMG_1199.jpg


I have posted this server before but recently added another external cage, its now up to 100TB running UnRaid 6.2 beta with several dockers with 45.7TB of useable space, 29 hard drives of varying space in total, using onboard SATA, a 16 port Areca and a SUPERMICRO AOC-SAS2LP-MV8 8 port controller.
 
Last edited:
I have posted this server before but recently added another external cage, its now up to 100TB running UnRaid 6.2 beta with several dockers with 45.7TB of useable space, 29 hard drives of varying space in total, using onboard SATA, a 16 port Areca and a SUPERMICRO AOC-SAS2LP-MV8 8 port controller.

do you have any vmware storage on this server? as in presenting some part of this storage to vmware as a datastore.
 
Orddie, UnRaid does support sharing via NFS to vmware/XenServer. Sadly.... unless you're putting that share on the "cache drive" (hopefully SSD based), you're limited to slightly lower IOPS than a single drive can provide.

With the current beta for UnRaid, you can have two dedicated parity drives for your data. This means that if you have a VHD on one of the shares, you're writing to three spinning drives at the same time.

I'm not sure of Ashman's implimentation and I hope he provides some details because that looks sweet!
 
Amount of total storage: 72TB

Chassis 1

48TB

Case Norco 4224
PSU Supermicro PWS-665-PQ
Motherboard Supermicro X9SCM-FO
CPU Core i3 2120
RAM 4 GB ECC
Controller Cards LSI 9260-4i
HP 3gbps SAS Expnader
Hard Drives 24 Toshiba DT01ACA200 2TB
Operating System Windows 7
OS Drive Adata SP600 128 GB SSD
Voltaire HCA 10Gbps Infiniband Card

Chassis 2

24TB

Case Norco 4020
PSU OCZ ZT 750
Motherboard SOme Acer board for power to the expander
Controller Cards Intel 6gbps RES2SV240
Hard Drives 12 Seagate ST2000DM001 2TB

Chassis Linked by SFF-8088 Cable.

20160731_193614 - Copy.jpg

20160731_193956 - Copy.jpg
 
Last edited:
Compared to most of you guys, I only have baby fileservers :(
(duplicate 16 x 8TB He8 setups in sig)
 
Woo hoo. New SuperMicro box for zfs nas.

Replaced my old Dell 2950 / MD1000 setup, as featured in this here thread:
https://hardforum.com/threads/27tb-zfs-with-dell-9th-gen-hardware.1898262/#post-1042274074

14TB + 40TB = 54TB raw.
Enjoy!

One pool is set up, the other 14 1tb drives aren't. Rough life.
v5Xa7ha.png


kfv9FvA.png

WNvCDQQ.png


Chassis. And my old shoe.
gv5c6u9.jpg


My fancy 120mm fan wall. Noctua fans are great. Also swapped out the 1200w redundant PSUs for a single 920w "SQ" super quiet power supply. It is indeed now super quiet.
DAYI72g.jpg


Front view looks like everyone else's. 4tb drives are lit. The rest are 1TB drives... Fancy paint can mounting system. I will rack this eventually. I broke the rails it came with. I swear I'm not an idiot.
tuCd8UL.jpg


4 x 500gb WD Black in Raid 10 for system disk. Looked forever to find the dual drive brackets.
nlSEP1T.jpg


Adaptec RAID card, Flashed IBM M1015 HBA. Cables are apparently the most expensive part of a build.
PQ6Ai81.jpg


Stack of 4tb WD Reds. Technology trash in background.
x4oko3p.jpg
 
working on new file servers 2x Dell 2100
CPU Xeon x5650 (2)
Ram 144gb
OS 512 850 Evo
Data WD Red 3tb (12) raid 6 w/BBU
Raid Card LSI 9260-8i




This is all 12 Reds in Raid Zero for Lulz
 
My very modest by comparison setup:

cwpOXEI.jpg


My trusty 5 year old Synology DS-1511+ with 5x2TB WD Reds in RAID 5, plus a 2TB+4TB in the external enclosure for backup. Also have the pfSense router/firewall, a 16 port Netgear switch, and a RPi running Ubuquiti UniFi for managing the APs.
 
Back
Top