large scale burst setups?

Discussion in 'Mining & Cryptocurrency' started by mkrohn, Jan 12, 2018.

  1. mkrohn

    mkrohn [H]ard|Gawd

    Messages:
    2,028
    Joined:
    Apr 30, 2012
    Yeah theres some burst threads... Lets just talk specifically about scaling burst rigs up here.

    I'm sitting here with a ryzen 7 with a bunch of internal drives and every USB3 port also plugged. What are my best options for ramping things up? I have an R710 home server but the C2100 has a lot more bays while staying super cheap.

    Should I buy one C2100 at a time and fill those up with drives for burst?
    external SAS boxes?
    Just keep building Ryzen 7's and max them out with both USB ports and internal drives?

    The ryzen if I wanted could fit 3 GPU's in and be useful for CPU mining so it can be multi purpose. The C2100 era CPU's suck for mining and the newer replacements are more expensive than they're worth. a ryzen 1700 easily plots twice as fast as dual Hex CPU's in my R710. My laptop CPU even plots as fast.

    Do I basically have 1 maxed out ryzen rig and have it mine every drive connected to it or do I go with network shares and process those over the network?
     
  2. cdabc123

    cdabc123 [H]ard|Gawd

    Messages:
    1,393
    Joined:
    Jun 21, 2016
    perc 6i cards with 8 cheap used sas drives each. put the drives in a pile and point a fan in the general direction
     
  3. mkrohn

    mkrohn [H]ard|Gawd

    Messages:
    2,028
    Joined:
    Apr 30, 2012
    don't the sas drives use more power than they're really worth? Also last I checked the 2Tb drives were the sweet spot for pricing which gets crazy when you try to scale it way up since 4tb drives cost a lot more (last i checked)
     
  4. cdabc123

    cdabc123 [H]ard|Gawd

    Messages:
    1,393
    Joined:
    Jun 21, 2016
    no idea on watts however the cheapest for tb are used sas drives at les then $10 a tb
     
    Last edited: Jan 12, 2018
  5. mkrohn

    mkrohn [H]ard|Gawd

    Messages:
    2,028
    Joined:
    Apr 30, 2012
    $10 a gb is terrible!
     
  6. cdabc123

    cdabc123 [H]ard|Gawd

    Messages:
    1,393
    Joined:
    Jun 21, 2016
    lol tb. 10$ a gb is like ddr4 ram levles.
     
  7. mkrohn

    mkrohn [H]ard|Gawd

    Messages:
    2,028
    Joined:
    Apr 30, 2012
    RAM prices? you mean getting rammed prices?
     
  8. w1retap

    w1retap [H]ardForum Junkie

    Messages:
    12,327
    Joined:
    Jul 17, 2006
    This reminds me.. I have a Highpoint Datacenter 7280 laying around. (32 drives on a single card) I wonder how good it would be for mining Burst.
     
  9. pclausen

    pclausen Limp Gawd

    Messages:
    339
    Joined:
    Jan 30, 2008
    My setup consists of a primary chassis with 24 hot swap bays and a server grade motherboard (SuperMicro X10 variety with a LGA 2011 V3 socket).

    On that motherboard I have 4x LSI 9211 HBAs. 1 of them has 8x 6Gbps SAS lanes that connects to the internal SAS2 expander backplane for the first 24 drives.

    The 3 additional HBA's each have 2 SFF-8088 connectors (total of 8x 6Gbps) per HBA. Each of these are connected to additional chassis with SAS2 expanders for an additional 24 drives each.

    So my total capacity is 96 drives. Right now I have 84 drives going. They are a mix of 2TB, 4TB and 8TB drives. Mostly WD REDs.

    Current capacity is 260TB and I can scan all my plots in less than 30 seconds.

    Since I have a total of 6 SFF-8088 external connectors, I can connect a total of 6 24 bay chassis to the main unit, for a total disk capacity of 168.

    I'm slowly upgrading all drives to 8TB ones, so this one server will eventually have 1.344 PB.

    Here are some pics:

    Primary chassis. CPU is E5-2683 (14 core / 28 threads)

    [​IMG]

    Whole stack of chassis:

    [​IMG]

    Here it is just after scanning all the plots for a new round:

    [​IMG]

    The key to fast plot scanning speed, is having plenty of bandwidth available to all your drives at once with no bottlenecks.

    The chassis I use are Supermicro 846 with SAS2 backplanes (this is very important as the older SAS backplanes do not support large 6TB and above drives). This guys has some on ebay right now for $399 plus shipping, which is not a bad deal:

    https://www.ebay.com/itm/Supermicro-846E16-R1200B-BAREBONE-4U-Server-BPN-SAS2-846EL1-24x-TRAYS-NO-PWS/382277186258?ssPageName=STRK:MEBIDX:IT&_trksid=p2060353.m2749.l2649

    The HBA's that I use run about $50 each for the externals and $75 each for the internal.

    Here are the internals for a really good deal:

    https://www.ebay.com/itm/LSI-9211-8...165520?hash=item1a3c616510:g:YgMAAOSw6DVaLFBw

    Externals:

    https://www.ebay.com/itm/LSI-SAS9200-8e-Dual-Port-External-SAS-6Gbps-RAID-Controller-Hi-Pro-H3-25260-02G/162819231677?hash=item25e8c84fbd:g:FbsAAOSwdnZaFidn
     
    Last edited: Jan 12, 2018
  10. w1retap

    w1retap [H]ardForum Junkie

    Messages:
    12,327
    Joined:
    Jul 17, 2006
    Nice. Do you have the 9211's in IT mode to monitor SMART parameters with any alerting if a drive starts to go bad?
     
  11. N4CR

    N4CR 2[H]4U

    Messages:
    2,188
    Joined:
    Oct 17, 2011
    What's your daily take after power?
    I'm considering grabbing more 8tb drives again could run 6 without much hassle.
    Nice rig..
     
  12. pclausen

    pclausen Limp Gawd

    Messages:
    339
    Joined:
    Jan 30, 2008
    Yes, the 9211's are in IT mode and flashed with the latest firmware.

    I don't really bother with monitoring SMART parameter's. If a drive fails, the burst miner will alert me and just drop that plot from its scan routine. Then I just pull the drive, plug in a replacement, plot it, and add it back to the miner config file.
     
    Last edited: Jan 13, 2018
    w1retap likes this.
  13. pclausen

    pclausen Limp Gawd

    Messages:
    339
    Joined:
    Jan 30, 2008
    Here's my take so far this month:

    [​IMG]

    So $733 at the current rate. So that should work out to about $1,500 for the month or about another 10 8TB drives worth. :)

    My burst server consumes 17.6 kWh a day and I pay about $0.10 per kWh before crediting for my solar production, so at face value, that works out to $1.76 per day, or $53/month.

    A lot of my drives (60 out of 84) are older power hungry Hitachi 2TB 7200 RPM drives. Once they are eventually replaced with power efficient WD RED's, power consumption should drop in half.

    The CPU in my burst server is also mining XMR at the same time is it mining burst, which brings in an additional $4/day. I paid $300 for that E5-2683 on ebay, so that has long since ROI'ed already. I'm thinking about getting a 2nd CPU and swapping in a dual socket motherboard.
     
    Last edited: Jan 13, 2018
    Jon855, N4CR and w1retap like this.
  14. Jon855

    Jon855 Avatar Fleshlight Destroyer - [H] Certified

    Messages:
    7,565
    Joined:
    Aug 25, 2005
    I would love to pick your brain here if I could. I'm sure you've seen my thread about me complementing a 480TB or at the very minimum I could start with a 240TB node. PM sent.
     
    Last edited: Jan 18, 2018 at 3:39 AM