Launching a 12 Card AMD RX580 Windows 10 based Mining Rig - Blog

Discussion in 'Mining & Cryptocurrency' started by Archaea, Oct 25, 2017.

  1. Eickst

    Eickst [H]ard|Gawd

    Messages:
    1,521
    Joined:
    Aug 24, 2005
    Yep I do one at a time, flash bios, test hash, remove card and start with next one. It takes longer to set up a rig, however the time saved troubleshooting one card/riser is a blessing.
     
  2. Airbrushkid

    Airbrushkid [H]ard|Gawd

    Messages:
    1,877
    Joined:
    Aug 27, 2007
    Sorry to hear you having problems. I have 3 rigs running 12 cards each. But the motherboards are all different and none of them are 12 boards. I am looking for an answer for you. I really think it's a bios problem but not for sure. Gotta look some more.
     
    Archaea likes this.
  3. Archaea

    Archaea [H]ardness Supreme

    Messages:
    6,913
    Joined:
    Oct 19, 2004
    Interesting
    I had that exact issue at first.
    But I noticed that the fast scrolling text indicated an issue with PCI-E 1 and PCI-E 2. So I shut down, unplugged both 1&2 cards, and rebooted and it just went to a black screen instead of scrolling the errors.
    I then swapped out those two card's risers and added each card back one back one at a time. I then had no more scrolling errors, but still just a black screen that never progressed.

    So I chalked it up to two more bad risers (5 total bad risers out of about 35 risers) - and then gave up - because with nothing more than a black screen what do you do? I logged in at simplemining.net but never saw my rig appear.



    I suspect the problem they allude to in that writeup is just a bandaid on a sword wound. Meaning the way I read it, you have something failing, and you are eliminating the error notification, rather than figuring out what is failing. The micropause issue seems VERY much relevant to what I am experiencing in Windows.

    I think I still have something failing - I don't know what yet. I wish I knew a way to specifically test each individual PCI-E riser/card. Like a quick cinebench type tool that you could run on each card individually and see what card/riser has issues.

    Airbrushkid, do you use auto, gen1, gen2, or gen 3 for your cards in your 12 card rigs?

    I noticed in another thread that now with 1709 Windows update, supposedly Windows 10 can run 12 Nvidia cards. Sure seems like 12, 1070 cards would be less trouble than the 12 RX580 -- at least from my vantage.
     
  4. thebufenator

    thebufenator Gawd

    Messages:
    573
    Joined:
    Dec 8, 2004
    Use gen2 for PCIe.

    Gen 1 has not worked for me on my 6 cards rigs, and gen 3 used too many lanes. This was on my Z270 boards, none of which are mining specific boards.

    A suggestion I saw on btc talk, was to use hot glue on the riser connections. Test each card with its bios and a riser, once it tests good, put hot glue on the riser-usb connections.Then once you have the cards situated on the board, hot glue the riser to the mobo pcie slot. It is removable, but it prevents vibration and such. I started doing it on my rigs and I thing it is helping.
     
    Archaea likes this.
  5. Airbrushkid

    Airbrushkid [H]ard|Gawd

    Messages:
    1,877
    Joined:
    Aug 27, 2007
    I run PCIE Auto and Simplemining OS. I don't like Windows for mining. But for gaming yes. Mine been running non stop for about month now. And for risers I've had 20 out of 100 go bad so far but I haven't tried them all. The riser, usb/pcie x1 adapter card and usb cable can go bad.


     
  6. thebufenator

    thebufenator Gawd

    Messages:
    573
    Joined:
    Dec 8, 2004
    As a sidenote, I have a 5x card RX rig that recently went stupid when I switched from ribbon risers to usb risers. I have had all kinds of stability issues that went away when I turned off the box fan blowing on the rig. I can only assume it was related to vibrations induced from the airflow affecting the risers. Now that it is stable for the first time in weeks I am afraid to shut it down to hot glue all the cables.
     
  7. Archaea

    Archaea [H]ardness Supreme

    Messages:
    6,913
    Joined:
    Oct 19, 2004
    FINALLY

    First time with all 12.

    I think I’ve still got a bad riser in there. Maybe on the fourth card left to right in wattman because it took me a half dozen restarts to get that one to switch to “compute mode” in the AMD drivers.

    BUT. They are all working at the moment

    A few notes:

    This project wasn’t at all worth the level of effort. I’ve literally spent probably 100 hours + on this one rig and it’s not quite right yet. I obsess about things when I get started and spend way too much energy on something like this. My wife literally started crying saying this was all I’d done for last two weeks straight when I was home from work. Not good.

    Unless you can tinker with something endlessly this project isn’t worth pursuit unless you get lucky -- with the state of current drivers, support, and lack of existing documentation out there to draw from. Maybe give it six months. I’ve found that the more GPUs one engages the more headache there is. The four to six GPU rigs seem to be pretty much problem free.

    Two 850 watt power supplies are insufficient for 12 RX580 cards

    I ended up adding a little 450 watt power supply to run two of the cards and the motherboard and the 450 watt power supply is insufficient. If the undervolt fails and wattman resets the 450 watt PSU fails. I’m single mining Eth right now with 1130 MHz GPU clock and 2100 MHz memory clock. Single mining ETH to save on power use since my PSUs can’t handle dual mining power draw. I think you’d need two 1200 watt PSUs or three 850 watt PSUs to have the safe power to spare if the undervolt fails.

    The windows UI is still buggy right now. I can’t click start button and I can’t launch IE. Remove all cards and the Windows UI issues disappear. I think this is the remaining bad riser at fault but I don’t know how to narrow down which one it is.

    I’ve had five out of 30 risers fail or DOA... and probably more temperamental. I’ve got five different kinds of risers. I can’t say any are any more reliable than the next brand.

    I’m still relatively new to the world of mining. Started dabbling in June, fired up more serious in late August with my first dedicated rig of eight 1080TI. Then a rig with eight 1060 and two RX 580, no a rig with 12 RX580.

    More to come.

    C8B6249B-7418-412C-982D-E232DB181BC2.jpeg
     
    Last edited: Nov 13, 2017
  8. thebufenator

    thebufenator Gawd

    Messages:
    573
    Joined:
    Dec 8, 2004
    I have found no need to run afterburner undervolting if the bios is modded for 900 voltage (0.900), and a clean ddu/reinstall of AMD drivers is done. If the bios is modded after drivers are installed, Wattman will keep the old voltages applied even if you never open it in the driver config.

    Something to consider. My measured power draw with a single card connected to a kill-o-watt was 110-115 watts on Eth going up to 145 watts dual Eth + lbry. That was with the mobo on an atx psu and the gpu being powered by a server psu + breakout board connected to the kill-o-watt.
     
  9. Airbrushkid

    Airbrushkid [H]ard|Gawd

    Messages:
    1,877
    Joined:
    Aug 27, 2007
    I'm happy your getting somewhere with your rig. I was going to build a AMD rig but after all the post and chats on the mining forums I'll stay with Nvidia they seem to be very easy to set up.
     
    Archaea likes this.
  10. thebufenator

    thebufenator Gawd

    Messages:
    573
    Joined:
    Dec 8, 2004
    What got it all running? Just swapping risers?
     
  11. Archaea

    Archaea [H]ardness Supreme

    Messages:
    6,913
    Joined:
    Oct 19, 2004
    Thanks for the input.

    What GPU clockspeed are you setting to 900v? I know where to set that in the BIOS, but didn't take it that low.
    900 volts is pretty low I thought. Are you finding you can consistently get a rack of cards that low, or are you individually setting voltages that low.

    Whatever GPU-Z shows for the GPU only draw, I think you can safely add 50 watts to that for the at the wall measurement. With dual mining My graphics cards are 130-140 watts in GPU-Z --- putting them at 180 to 190 watts from the wall.

    That's with the clock speed limited to 1130 in wattman (which does reduce the voltages automatically) - but no voltage adjustments otherwise done manually.
     
  12. thebufenator

    thebufenator Gawd

    Messages:
    573
    Joined:
    Dec 8, 2004
    I can consistently run at 900 with the core around 1130. Some are fine at 1155 but it makes no difference on Eth speed. GPU-z / afterburner consistently show ~880-890 when set to 900.

    Yeah I don't go by what GPU-z shows for power draw. Like I mentioned, I had a single RX card plugged into a killowatt, so that was nice to test voltage changes and clockspeed.
     
    Archaea likes this.
  13. Archaea

    Archaea [H]ardness Supreme

    Messages:
    6,913
    Joined:
    Oct 19, 2004
    There's just more moving parts with AMD. I think once you get it squared away they can be very good. But until you update the BIOS your only going to see about 19-21MH/s with a RX580 at full tilt. Now they've got computer mode you need to enable in the driver control panel to get significantly more MH/s -- but you can only enable it one card at a time, and have to restart the AMD driver control panel again each time. I'm not convinced Wattman is reliable about retaining settings - sometimes it does sometimes not - and I don't know what triggers it to lose settings with reboots/updates. afterburner is hit or miss with AMD on whether you can even control the AMD cards. The current afterburner 440 can't even control my AMD cards with the 17.11.1 drivers.

    Getting a good AMD bios is half the initial battle. I made the bios for these cards I'm using by pirating some of the timings for other cards and using a few tips off anorak.tec That's all a bit of learning curve too, but I think I have the basics down now.
    Once you get the AMD rigs going they seem to be very stable. I've had two RX580 going for over a month without crashing one time on my 10 card rig - but the 1060's that were on the rig with it have crashed several times over that span. (two instances of nice hash running - one for eight 1060's, one for two RX580s -- same 1200 watt power supply shared for all 10 cards)

    But all in all, Nvidia's run great right out of the gate - turn them to 65% power target, overclock memory to just shy of 600Mhz, and return clocks to factory speed using the GPU Mhz boost and your off to the races. >= ~100% stock performance at 65% power draw - overall very reliably too.
    Other bonus for Nvidia is that afterburner has always worked with Nvidia (that I've seen) and it's very reliable in keeping the power draw down to that 65% target.

    If I was going to repeat this project it'd be with 12 1070 cards instead now that I read Nvidia also supports 12 cards in Win 10. That wasn't a known thing just a few weeks back when I started this thread.
     
  14. Archaea

    Archaea [H]ardness Supreme

    Messages:
    6,913
    Joined:
    Oct 19, 2004
    yes.
    Apparently I had at least four intermittently bad riser cards of 12 -- and I have at least one more I haven't found out.
    Simplemining OS helped me root out those first two slot cards that I hadn't been able to determine with windows with that scrolling PCI-E error issue, but it can't seem to help me ferret out this last one, because it's just a black screen now - with no progression beyond.

    I'm hoping to just let the eth mining run until one of the cards fails - that'll be my suspect riser replacement.
    if hashrate just slows - who knows -- cause I was dealing with that before and I'd swap out the riser with the hash rate that slowed to like 6MH/s and it wouldn't fix the problem.
     
  15. Eickst

    Eickst [H]ard|Gawd

    Messages:
    1,521
    Joined:
    Aug 24, 2005
    Yea I've found while they don't have the same high hash rates as modded rx580's, the 1060/1070 line sips power and is way easier to set up. All of the rigs I've been building lately have been with nvidia cards, running on nvOC. Super simple
     
  16. Airbrushkid

    Airbrushkid [H]ard|Gawd

    Messages:
    1,877
    Joined:
    Aug 27, 2007
    I know guys out that getting 30 to 31 MHs with 1070's.
     
  17. Archaea

    Archaea [H]ardness Supreme

    Messages:
    6,913
    Joined:
    Oct 19, 2004
    anyone following along and interested in a strong PSU for a build like this - I saw this today.
    https://www.newegg.com/Product/Product.aspx?item=N82E16817152059

    RaidMax Vampire 1000 watt Gold Rated PSU for $60 after rebate at newegg.

    That's unbeatable. I bought two more for my mining operations. (I already had one of these PSU's and was very happy with it - it held up pulling 1050 watts from the wall for days on end until I balanced the load out more with another PSU).
     
  18. collegeboy69us

    collegeboy69us [H]ardness Supreme

    Messages:
    5,025
    Joined:
    Jul 27, 2003
    Just a heads up -- my experience with RaidMax isn't that great, I had an 850W psu of theirs... the cables themselves became insanely brittle after about 1 year. To the point where if you moved the cable the PVC plastic covering individual wires cracks and breaks off, in addition, the copper wire inside started to oxidize and basically fell apart. Their customer service is useless if you just want to buy another set of cables. Because of this I'm now 100% a corsair man in terms of PSU's (their support is unbeatable) recently had one of theirs die and take a mobo out (cheap worthless board) they are sending me a free upgrade and a check for $60 to cover the damage :)