[H]ard Forum Storage Showoff Thread 2015

Discussion in 'SSDs & Data Storage' started by EnderW, Jan 1, 2015.

  1. xx0m3gaxx

    xx0m3gaxx [H]Lite

    Messages:
    69
    Joined:
    Oct 24, 2013
    What drives do you currently have in the system?
     
  2. pclausen

    pclausen Limp Gawd

    Messages:
    454
    Joined:
    Jan 30, 2008
    Made some upgrades over the last few week. Got 120TB in primary server and 120TB in backup server.

    Primary is the top 2 chassis in the below stack. Just got a 24 bay SuperMicro 216 2U chassis that is connected the the main Areca controller in the 846 chassis below it. Starting to fill it with Samsung 840 Pros for "scratch" space.

    [​IMG]

    The lower 3 chassis (826, 846, 846) is the backup server.

    Primary storage setup as follows:

    [​IMG]

    [​IMG]

    Secondary storage setup as follows:

    [​IMG]

    I need to upgrade the LSI 9266 controller to something with more SFF-8087 ports so I can get rid of the bottleneck caused by cascading 48 of the 2TB drives over a single 4 channel SFF-8087 link. I'll probably pick up another Areca 1882 like what I have in the primary server.

    Backside of the SuperMicro stack. Primary server has 10GigE to the core switch and there is a direct server to server 10GigE link as well.

    [​IMG]

    A shot of the switches and dual GigE Fiber ONTs:

    [​IMG]
     
    {NG}Fidel likes this.
  3. guc32

    guc32 n00bie

    Messages:
    62
    Joined:
    Apr 27, 2006
    Thats some fine gear you have there!!!
     
  4. yoyostile

    yoyostile n00bie

    Messages:
    3
    Joined:
    Jan 14, 2016
    Right now 2x Sandisk 120gb SSDs as zfs mirror for proxmox and 3x3TB & 3x4TB WD Red als BTRFS Raid. I want to switch to ZFS RAIDZ2 soon, so I'll need some more 4TB Reds... :)
     
  5. rkd29980

    rkd29980 Limp Gawd

    Messages:
    166
    Joined:
    Oct 19, 2015

    I hope this isn't too bothersome for you but I am building my own array and am looking to learn more about the networking side of it. I noticed your network switches and all the cables connected to your servers and am hoping you would be willing to explain your particular setup and how someone like me can duplicate your setup and transfer speeds. I have 4 storage servers that will make up my array.
     
  6. unholythree

    unholythree Limp Gawd

    Messages:
    200
    Joined:
    Sep 23, 2011
    The bottom two 4U server chassis just hold disks, a power supply, and an SAS expander if I had to guess. I think the actual server is the 2U chassis above them; you can see the I/O plate and motherboard connectors.

    His setup is likely something like this.
    Server with HBA/RAID card -> Internal SAS 8087 cable -> 8087 to 8088 adaptor -> External SAS 8088 cable -> 8087 to 8088 adaptor (in the first DAS/disk shelf/expander chassis whatever you call it) -> 8087 cable -> to expander card going to disks and out to the next chassis...

    You can buy chassis like his premade or build your own. A cheap and popular option is the Rackables se3016 several are shown in this thread.

    Now if you wanted to make some kind of storage cluster, that would be a different beast all together.
     
  7. WoooT

    WoooT n00bie

    Messages:
    40
    Joined:
    Aug 1, 2010

    Hi thinking about building something like that
    especially with the Fractal design case. How is the temperture on the disks?
     
  8. Dssguy1

    Dssguy1 n00bie

    Messages:
    10
    Joined:
    Mar 24, 2016
    I am a total nub to all of this stuff but everybody has to start somewhere right? I just bought a Rackable 3016 off Ebay for $130 and planning to run a server off my HP Z800 to access the drives. Quick question about hard drives from you guys who are seriously awesome at this stuff. I have been snagging cheap Hitachi Ultrastar 7k3000 SAS (2 TB and 3 TB) drives off eBay. Most of the time they are direct Hitachi Refurbs in original sealed packaging and ranging from $35-60 each. So far I have 6 x 2TB and 5 x 3TB.

    I haven't seen very many people using Hitachi drives and when they do, they are normally deskstars. Am I going down a path I don't want to be down? Either with the SAS drives or with Ultrastars in general?
     
  9. iamwhoiamtoday

    iamwhoiamtoday Limp Gawd

    Messages:
    493
    Joined:
    Oct 29, 2013
    I have 12 3TB drives, split between seagates and Ultrastars. Been using them for the last 2-3 years without a single drive failure. The $70 3TB refurbs have served me pretty well so far. I'd highly recommend them as a cheap way to expand storage. Just make sure to have both redundancy and backups, so that *when* a drive or three fails, you still have all of your data.
     
  10. bds1904

    bds1904 Gawd

    Messages:
    1,001
    Joined:
    Aug 10, 2011
    I have over 300 ultrastar's in service, Various models and sizes. I have been very happy with the performance and failure rate so far. I have them in several se3016 enclosures that run 24/7/365 as well as several synology NAS units. All of them stay at good temps and have good power.

    I have only suffered 3 failures so far in 3.5yrs. All 3 have been different batches and different models. All 3 failures were also healed by hot-spares with no issues.

    Post failure inspection yeilded an unacceptable amount of read errors caused by bad sectors. The usual bad-hdd result IMHO.
     
  11. IceDigger

    IceDigger [H]ardForum Junkie

    Messages:
    10,351
    Joined:
    Feb 22, 2001
    Currently I'm just using SAS drives rather than SATA drives in my server. The costs aren't that much more for the 2tb to the 4tb. The 5tb and up SAS drives are crazy expensive compared to the SATA though.
     
  12. ITGuy1024

    ITGuy1024 n00bie

    Messages:
    44
    Joined:
    Jan 22, 2016
    Nothing crazy. 9TB FreeNAS server I user for home. Running strong for 2 years now, no drive failures.
     

    Attached Files:

  13. guc32

    guc32 n00bie

    Messages:
    62
    Joined:
    Apr 27, 2006
    Thanks for info about Ultrastars. I didnt know about them and now I see they are enterprise drives. :D
     
  14. alex_di

    alex_di n00bie

    Messages:
    35
    Joined:
    Mar 15, 2016
    What's prompting the migration from BTRFS?
     
  15. Dephcon

    Dephcon [H]ard|Gawd

    Messages:
    1,234
    Joined:
    Jul 27, 2004
    How do you like the USG?
     
  16. Iron67

    Iron67 n00bie

    Messages:
    28
    Joined:
    Oct 4, 2008
    My media server (update 3)

    TOTAL STORAGE: 52 TB (34.7 usable)

    Chassis: Supermicro CSE-846XE26-R1K28B
    Mainboard: Asus M2N32-SLI Dlx/WF
    CPU: AMD Athlon64 X2 6000+
    RAM: 2x Samsung PC2-6400 2GB
    VGA: Asus EAX1950XTX
    HBA: Adaptec RAID 5805ZQ
    Cache: Samsung SSD 850 Pro MZ-7KE256BW
    HDD: 2x Seagate 7200.10 ST3250620AS
    HDD: 2x Seagate 7200.11 ST31000340AS
    HDD: 2x Western Digital Caviar Black WD1001FALS
    HDD: 8x Western Digital RE4 WD2003FYYS
    HDD: 8x Western Digital Re WD4000FYYZ
    ODD: LiteON iHAS 120
    DTV: Hauppauge HVR-2200
    FAN: Original PWM fans with thermal probe & two channel controller
    OS: Windows 7 Ultimate

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]
     
    Last edited: Apr 16, 2016
  17. Fanatic_Joker

    Fanatic_Joker n00bie

    Messages:
    9
    Joined:
    Mar 14, 2013
    Which kind of SAS Expander you are using ?
    I know LSI SAS... but which one...

    Nice setup by the way
     
  18. Blue Fox

    Blue Fox [H]ardForum Junkie

    Messages:
    11,628
    Joined:
    Jun 9, 2004
    They're using the LSI 6gbit one that's integrated into the backplane of the chassis. Chip on it should be the SAS2x36.
     
  19. Iron67

    Iron67 n00bie

    Messages:
    28
    Joined:
    Oct 4, 2008
    Exactly, the p/n is BPN-SAS2-846EL2 and is based on LSI SAS2X36.
     
  20. Iron67

    Iron67 n00bie

    Messages:
    28
    Joined:
    Oct 4, 2008
    My media server (update 4)

    TOTAL STORAGE: 52 TB (34.7 usable)

    Chassis: Supermicro CSE-846XE26-R1K28B
    Mainboard: Asus Crosshair V Formula Z
    CPU: AMD FX-8370
    RAM: 2x Crucial Ballistix Elite DDR3-1866-8GB
    VGA: Asus EAX1950XTX
    HBA: Adaptec RAID 5805ZQ
    Cache: Samsung SSD 850 Pro MZ-7KE256BW
    HDD: 2x Seagate 7200.10 ST3250620AS
    HDD: 2x Seagate 7200.11 ST31000340AS
    HDD: 2x Western Digital Caviar Black WD1001FALS
    HDD: 8x Western Digital RE4 WD2003FYYS
    HDD: 8x Western Digital Re WD4000FYYZ
    ODD: LiteON iHAS 120
    DTV: Hauppauge HVR-2200
    FAN: Original PWM fans with thermal probe & two channel controller
    OS: Windows 7 Ultimate

    [​IMG]
     
  21. Trimlock

    Trimlock [H]ardForum Junkie

    Messages:
    15,119
    Joined:
    Sep 23, 2005
    What are you using that x1950 for?
     
  22. Iron67

    Iron67 n00bie

    Messages:
    28
    Joined:
    Oct 4, 2008
    I use it for dual monitor.
     
  23. Sovereign

    Sovereign 2[H]4U

    Messages:
    3,080
    Joined:
    Mar 21, 2005
    Included links for reference. Not paid to endorse, advertise, or promote any of the products linked.

    Total Advertised Capacity: 52TB
    Total RAID6 Capacity: 44TB
    Current Setup: 20TB RAD6, 16TB RAID6 (see shenanigans* below)

    Case: Cooler Master Storm Trooper
    HDD Expander: FatCage 5-in-3
    PSU: eVGA SuperNova G2 1300w
    Motherboard: ASUS P8P67 Deluxe B3
    CPU: Intel i7 2600k @ 4.4GHz
    RAM: G.Skill Sniper DDR3-2400 4x8GB
    Controller Cards (if any): LSI MegaRAID 9260-8i w/BBU and Intel RAID Expander
    Hard Drives (include full model number): HGST NAS 4TB, 13x - H3IKNAS40003272SN (0S03664)
    Operating System: Windows 10 Pro x64 on Samsung 850 EVO 1TB

    I am nowhere near full--the original 5x4TB RAID6 gave way to 7x, then I saw the drives mentioned above on sale (lowest price they've been in years and don't go below that according to the trends I could find). Decided to max out.

    Note that if you want to stick a FatCage in the Storm Trooper you have to bend or dremel. From the photos I saw it didn't look like it had tabs in the 5.25" drive bays but it does.

    * I have to do some shenanigans. When I was new to RAID (level 5, PERC5i, 3x2TB drives) I figured 4kB cluster sizes were fine and dandy. Except when I went to move from 5x to 7x 4TB drives, things weren't fine and I ended up with a two part dynamic disk since I hit a filesystem limitation.

    Now I'm copying everything off the 20TB RAID6 onto the 16TB RAID6. Then I'm going to do an online capacity expansion to get to the full 44TB RAID6. I don't project needing drives again unless a unit dies outside warranty.

    All but some replaceable files are on Backblaze.

    This is/was my "everything" machine: games, storage, media, work (from home), HTPC. It also weighs over 100 pounds due in large part to the drives (the empty case is 31.7 pounds by itself). I am literally going to a hardware store to buy a hand cart next time I need to move it. LANs will use a "battle box" from now on to save me the hassle.

    [​IMG]
    MegaRAID Storage Manager

    [​IMG]
    Case Front (note the FatCage)

    [​IMG]
    Case Side (due to design of case, the drives themselves cannot be seen)
     
  24. ashman

    ashman Gawd

    Messages:
    805
    Joined:
    Mar 28, 2011
    IMG_1195.jpg IMG_1199.jpg

    I have posted this server before but recently added another external cage, its now up to 100TB running UnRaid 6.2 beta with several dockers with 45.7TB of useable space, 29 hard drives of varying space in total, using onboard SATA, a 16 port Areca and a SUPERMICRO AOC-SAS2LP-MV8 8 port controller.
     
    Last edited: Jun 22, 2016
  25. Orddie

    Orddie 2[H]4U

    Messages:
    2,211
    Joined:
    Dec 20, 2010
    do you have any vmware storage on this server? as in presenting some part of this storage to vmware as a datastore.
     
  26. iamwhoiamtoday

    iamwhoiamtoday Limp Gawd

    Messages:
    493
    Joined:
    Oct 29, 2013
    Orddie, UnRaid does support sharing via NFS to vmware/XenServer. Sadly.... unless you're putting that share on the "cache drive" (hopefully SSD based), you're limited to slightly lower IOPS than a single drive can provide.

    With the current beta for UnRaid, you can have two dedicated parity drives for your data. This means that if you have a VHD on one of the shares, you're writing to three spinning drives at the same time.

    I'm not sure of Ashman's implimentation and I hope he provides some details because that looks sweet!
     
  27. ashman

    ashman Gawd

    Messages:
    805
    Joined:
    Mar 28, 2011
    I have a sperate VMWare server with its own storage so I am not using this in any way with that environment.
     
    iamwhoiamtoday likes this.
  28. Orddie

    Orddie 2[H]4U

    Messages:
    2,211
    Joined:
    Dec 20, 2010
    thanks!
     
  29. Master_shake_

    Master_shake_ Little Bitch

    Messages:
    7,060
    Joined:
    Apr 9, 2012
    Amount of total storage: 72TB

    Chassis 1

    48TB

    Case Norco 4224
    PSU Supermicro PWS-665-PQ
    Motherboard Supermicro X9SCM-FO
    CPU Core i3 2120
    RAM 4 GB ECC
    Controller Cards LSI 9260-4i
    HP 3gbps SAS Expnader
    Hard Drives 24 Toshiba DT01ACA200 2TB
    Operating System Windows 7
    OS Drive Adata SP600 128 GB SSD
    Voltaire HCA 10Gbps Infiniband Card

    Chassis 2

    24TB

    Case Norco 4020
    PSU OCZ ZT 750
    Motherboard SOme Acer board for power to the expander
    Controller Cards Intel 6gbps RES2SV240
    Hard Drives 12 Seagate ST2000DM001 2TB

    Chassis Linked by SFF-8088 Cable.

    20160731_193614 - Copy.jpg
    20160731_193956 - Copy.jpg
     
    Last edited: Jan 30, 2017
  30. WhoBeDaPlaya

    WhoBeDaPlaya 2[H]4U

    Messages:
    2,575
    Joined:
    Dec 16, 2002
    Compared to most of you guys, I only have baby fileservers :(
    (duplicate 16 x 8TB He8 setups in sig)
     
  31. deadlift1

    deadlift1 [H]ard|Gawd

    Messages:
    1,287
    Joined:
    May 8, 2009
    Duplicate 128TB fileservers is not considered "baby" in my book.
     
  32. med

    med n00bie

    Messages:
    11
    Joined:
    Mar 3, 2016
    Woo hoo. New SuperMicro box for zfs nas.

    Replaced my old Dell 2950 / MD1000 setup, as featured in this here thread:
    https://hardforum.com/threads/27tb-zfs-with-dell-9th-gen-hardware.1898262/#post-1042274074

    14TB + 40TB = 54TB raw.
    Enjoy!

    One pool is set up, the other 14 1tb drives aren't. Rough life.
    [​IMG]

    [​IMG]
    [​IMG]

    Chassis. And my old shoe.
    [​IMG]

    My fancy 120mm fan wall. Noctua fans are great. Also swapped out the 1200w redundant PSUs for a single 920w "SQ" super quiet power supply. It is indeed now super quiet.
    [​IMG]

    Front view looks like everyone else's. 4tb drives are lit. The rest are 1TB drives... Fancy paint can mounting system. I will rack this eventually. I broke the rails it came with. I swear I'm not an idiot.
    [​IMG]

    4 x 500gb WD Black in Raid 10 for system disk. Looked forever to find the dual drive brackets.
    [​IMG]

    Adaptec RAID card, Flashed IBM M1015 HBA. Cables are apparently the most expensive part of a build.
    [​IMG]

    Stack of 4tb WD Reds. Technology trash in background.
    [​IMG]
     
  33. Irockzz

    Irockzz n00bie

    Messages:
    35
    Joined:
    May 27, 2016
    working on new file servers 2x Dell 2100
    CPU Xeon x5650 (2)
    Ram 144gb
    OS 512 850 Evo
    Data WD Red 3tb (12) raid 6 w/BBU
    Raid Card LSI 9260-8i
    [​IMG]

    [​IMG]

    This is all 12 Reds in Raid Zero for Lulz
    [​IMG]
     
    ArrowToTheKnee likes this.
  34. deadlift1

    deadlift1 [H]ard|Gawd

    Messages:
    1,287
    Joined:
    May 8, 2009
    Nothing like the kitchen countertop for building and testing.
     
  35. Irockzz

    Irockzz n00bie

    Messages:
    35
    Joined:
    May 27, 2016
    yep lol here they are now

    [​IMG]
     
  36. Ranulfo

    Ranulfo [H]ard|Gawd

    Messages:
    1,263
    Joined:
    Feb 9, 2006
    Ash, what case is that?
     
  37. EspoNation

    EspoNation Gawd

    Messages:
    781
    Joined:
    Sep 28, 2011
  38. Ranulfo

    Ranulfo [H]ard|Gawd

    Messages:
    1,263
    Joined:
    Feb 9, 2006
  39. Bandalo

    Bandalo 2[H]4U

    Messages:
    2,687
    Joined:
    Dec 15, 2010
    My very modest by comparison setup:

    [​IMG]

    My trusty 5 year old Synology DS-1511+ with 5x2TB WD Reds in RAID 5, plus a 2TB+4TB in the external enclosure for backup. Also have the pfSense router/firewall, a 16 port Netgear switch, and a RPi running Ubuquiti UniFi for managing the APs.
     
    spencers likes this.
  40. WhoBeDaPlaya

    WhoBeDaPlaya 2[H]4U

    Messages:
    2,575
    Joined:
    Dec 16, 2002
    Older pics of my duplicate fileservers. Specs in sig - 2x 128TB of goodness :D


    [​IMG]

    [​IMG]