Graphics Card Necromancy: Resurrecting a Dead GTX 690

If you guessed that the suspect component is the low side mosfet (Q15) to the right of the choke in the photo, you guessed right! We can even see that the short is in the upper right corner, as that's where the alcohol is evaporating off of. We can confirm that this our problem by removing the suspect component from the board, and checking the resistance across it, and the resistance of our circuit without it, which should be back to normal.

Edit: Well, I should probably clarify that that we can confirm that this is a problem - it's totally possible that we have more than one. Also, I checked to make sure that 0 ohm resistor they seem to be using as a fuse is still ok. It is.

With the mosfet in place, we have 1.6 ohms across it.


And here's our pad where it goes, with the mosfet removed.


With the mosfet removed, our resistance is back in the normal range.


And the resistance across our suspect mosfet, which should be several thousand ohms, is actually almost zero. This was very difficult to photograph. I think the real resistance may be more like 2 ohms, and one of the probes slipped, but you get the idea.


Luckily, we have a brand new replacement, which I purchased a few weeks ago for one of the other cards, and didn't end up using it.


Resistance across this new one is much higher. I tried to get a photo of this, but I need at least two more hands for that. Suffice to say that it's several million ohms.

 

Here's the new mosfet, soldered to the board in place of the old one.



Resistance across the circuit still checks out, at 7 or 8 ohms. I've got continuity to the gate pin on the controller, continuity to ground on the source pins, and continuity to the positive side of the circuit on the drain.

I sat for a little while, trying to think of anything else I should test before plugging the card into the system and powering it up. I triple checked the resistance on all three of the mosfet's terminals, and compared them to Card B, which still has this circuit unaltered. They're all within a couple of percent of each other. So, I plugged the card in, crossed my fingers, and hit the switch...













And after a brief eternity (probably less than 10 seconds), it worked!


Now that I know it at least kinda works, I need clean the flux residue off of it, and put the cooler back on. Then I can let it run long enough to check the voltages at the rail's controller. It's possible for it to run, and still be broken, such that it would fail again if I ran it for too long, or too hard.

 

Alright, posting from the test bench now, actually using the card. All the requisite voltages are there, and within spec.

The card seems to work, with the exception that GPU #2 (as reported by precision) runs really hot, so I bailed out of Heaven after about 30 seconds. It got up to about 80 before GPU #1 got past 55 or so.
Is the stock cooler on this card just that ineffective? I tried re-pasting it, but that didn't make any difference. I guess I should try one of the Arctic Cooling coolers I have piling up in my house. Kind of a bummer - they sure don't look as cool as the stock one.

Still, I just played a few minutes of Mass Effect 2, and everything seems to work. I redid the graphics settings and reinstalled the motherboard's chipset drivers, and now it runs at a steady 60FPS (with vsync on).

 

Thanks for your insight!

I think I may have figured it out, and this is a new one for me. It appears that my #2 GPU die may be slightly convex, and thus doesn't make very good contact with its heatsink. This seems to be exacerbated by the relatively low mounting pressure of the stock heatsinks, and I suspect also that the heatsink I was trying to use may also be slightly convex, which aggravates the problem.

This is the heatsink for GPU #1. See how the grease looks pretty even?


This is the heatsink for GPU #2. See the thin spot in the center?


I swapped one of the Arctic Cooling "Twin Turbo 690" coolers onto the card, and now both cores hit about 60 degrees after a couple of minutes of Heaven. GPU #2 still seems to run a couple of degrees hotter, though, and heats up faster, hence why I think it's the die surface, and not just the heatsink that's convex.

I had not considered that it might be running extra voltage on GPU number 2, but I'll have to check on that tomorrow.

 

Ok, so, good news and bad news today. Bad news is, there's still something wrong. Good news is, i'm not really convinced that the problem is the card. I managed to kill the card again, in the same failure mode as yesterday, where it burned out that little 0 ohm resistor AND the low side mosfet, probably not in that order.

Out of an abundance of caution, I stuck a thermocouple directly on the mosfet before running it, and I discovered that it gets alarmingly hot - over 100 degrees as measured, meaning Tj is likely much higher. I wasn't quick enough shutting it down, and it overheated the mosfet, which in turn led to overdrawing current through that little resistor. It's a good thing that resistor is there. I replaced both MOSFETs and the resistor, and the card now works again.

Now, one of the things that can cause a MOSFET to run hot is inadequate gate drive voltage, as I've talked about before. The gate drive is the signal that causes the mosfet to switch on. If the voltage is too low, the MOSFET will only switch on partially, and have a high resistance from the source (connected to ground) to the drain (connected to the load - our GPU in this case). This high resistance causes more of the power passing through the MOSFET to be dissipated as heat that you'd normally see, causing the MOSFET to run hot, which is undesirable for several reasons.

So. One thing I've noticed is that the voltage making it to the controller seems to be a little low. I had assumed this was intentional, as the card has a VRM that seems to exist for the purpose of regulating the 12V input power from the PSU that outputs about 11.5volts. There are even traces on the board to supply 12V power direct from one of the 8 pin connectors that are not used, that VRM is used instead. If you look at the photos I posted of card C, there are empty pads next to that 0 ohm resistor. They connect to the .95V rail's input power on one side, and the leftmost 8 pin on the other - to use it, I'd need to install another 0 ohm resistor on those pads, but I assume nvidia left this out, and included a VRM for a reason.

With this in mind, I'm starting to suspect that the reason my cards keep dying is not a defect in the cards themselves, but rather that the power being supplied to them by the power supply is insufficient, and some quirks of this .95v rail design cause the gate drive voltage to droop with the power supply's 12V voltage. I have a feeling that, under load. the "12V" being supplied by the power supply is dropping to like 11.5 or less. Notably, this power supply is an older 650W unit. 650W is the minimum that nvidia calls for for the GTX 690. Thus, I think the next step is to swap that unit out with a newer, more robust one. I have one I could borrow from my main rig, but it would be a huge pain to extract it with all of its cables, so I'm considering buying another.

Keeping in mind that I'm going to hook this thing up to used hardware I bought from ebay and repaired myself using questionable logic and methods, I'd prefer not to spend a gazillion bucks on something super fancy. Anyone have any recommendations on a reliable, but inexpensive power supply in the 1000ish watt range?

Edit: I did check the voltage, as Azrak suggested - GPU #2 is actually running a hair less voltage, I assume because it's running hotter and GPU Boost is throttling back a little.

 

Yes, I've considered that. The thing is, the rail that keeps failing doesn't seem to be connected directly to the motherboard power. Power comes into the card at three places. One is the PCI-E connector, and the other two are the 8 pin power connectors. The VRM I mentioned that produces "12 volts" doesn't connect to the PCI-E power at all, as far as I can tell, and instead is connected to one of the 8 pins*.

So, I've considered it, but it seems less likely to be the root cause of the problem than the power supply. Not impossible, though.

*I think. I've slept since I checked on this, but I'll double check it again tonight.

 

It actually won't post without the 8 pins connected. The VCore VRMs have a total of five phases each, three of which are connected to one of the 8 pins, and two of which are connected to the other. Without both 8 pins hooked up, the Vcore VRM doesn't power up at all.

Also, the .95 rail seems to be hooked up to both of the 8 pin connectors, such that it wants both to be working before it starts.

I've pretty much resigned myself to pulling my water cooled 1080 Ti out of my gaming machine, and testing with that, since it has a really nice Corsair 1000 watt power supply, with voltage monitoring. It's just that draining the coolant loop is a pain...

 

Correction to this: The system doesn't post, but the card does produce a nice, friendly warning on the screen about how you need to plug in the power connectors if you try to use it without them.


Anyway, good news, kids.

After some deliberation, I splurged, and hooked myself up with this beast - a Corsair AX1200i:


I know it's overkill in the overall wattage sense for my specific use case here, but I have some plans for other stuff in the relatively near future where I think having a power supply this robust will be handy, so I think it's worth it. Also, I really like the per-rail voltage monitoring that my RM1000i in my gaming machine has, and this unit really goes ham with that feature. It even lets you set individual overcurrent protection for each of the 6/8 pin cables, which is kinda neat. For troubleshooting faulty components, that seems like a really handy feature.

Anyway, I hate to say it, but I'm not super impressed with it. It appears that the components inside it are packed in there so closely that one of the ATX power connectors for the motherboard has a cap in the way of its little retaining clip, such that I can't quite get it plugged in securely (see the second photo). This isn't a huge deal for my case, since I'm not installing it permanently anywhere, but it would leave a pretty unpleasant taste in my mouth if I were using this in a normal build. I may yet return it as defective, but I haven't decided if I really care to deal with that. It does work as intended otherwise.

Nevertheless, I swapped out the old suspect power supply with this new hottness one, and now my low side mosfet runs way cooler. Instead of hitting 60C immediately when turned on, it hits about 40. Seems like maybe the gate drive voltage was, in fact, the problem.


Under load, it goes up to the 80s, but I still eventually get a black screen if I run the card under full load long enough. It doesn't seem to related to the .95 rail, though, as the card turns right back on if I turn the system back on, with no apparent damage. I have my doubts about the effectiveness of the Arctic cooling heatsink at cooling the VRMs, though, so it may be related to that, or the PSU's overcurrent protection not being set right. That seems unlikely, since the limits are something like 23 amps per cable (which is almost 300 watts per cable), but maybe I didn't read the instructions right. Still, i think it's a victory that our .95 rail seems to survive actual use, now.

One thing I may try is swapping the nvidia heatsink back on. The Arctic Cooling heatsinks have great big monster heatsinks for the core, but just a thin piece of aluminum for all the other parts of the card, unlike the nvidia heatsink, which has a massive casting for the memory and VRM, with some actual fins. The VRMs on this card are all somewhat more stressed than on a 680, because they have generally fewer phases in order to fit them all on there, and it wouldn't surprise me if the Arctic Cooling heatsink doesn't really cool them well enough.

 

Is that the cap you're talking about? I can desolder it later and check the capacitance, but it's probably best to get known good ones, if you can. I guess I could send you one if you can't, though. Are you in the US?


Can you clarify, you have two cards? One that used to work, but you removed this cap, and now it doesn't ("Board A"), and another where it was broken off, and never worked("Board B")?

I'd expect that the card would mostly work without it, as it seems to be there for the purpose of smoothing out noise in the 12V supply power, but there's a lot about this card that seems less robust than it maybe should be, to me. If you replaced it with a capacitor of some unknown value, though, one never knows what might happen.

PS: How did you manage to break the cap in the process of removing it? Are you just using a soldering iron, and not hot air?

 

Oh. Yeah, you really need hot air or at least hot tweezers to do this sort of SMD rework.

I desoldered the cap from one of my dead boards, and I measure it at ~11.8 uF. If I had to guess, I'd say it's probably supposed to be 12uF.

I guess I should post an update about my own Card D while I'm here.

I cobbled together one of the nVidia coolers such that it cools Card D's second GPU a little better, and then installed the system in the case, plugged it into my TV and played a few rounds (that is - several hours) of Mass Effect and Skyrim. The card seems to do fine, although the GPUs only seem to be seeing about 40% load, I think because the CPU in my test rig isn't fast enough to keep up with them. So, I really need to get my hands on a better motherboard/CPU, I guess, but it hasn't died yet, and it doesn't crash.

I unplugged it while I was vacuuming yesterday, but here it is as it sits currently.



My next order of business may be to modify the case so I can use it as a test bench. The short version of that being removal of the drive cage and cutting a hole in the bottom so a riser cable can reach out of the case.

 

Ok, so philosophical forum usage question:

I mentioned before that I bought a pile of other cards to work on, which are not GTX 690s. Would it be considered bad form to start a fresh thread about those? I'm thinking I may make some videos for Youtube, which I'd like to post here for the sake of sharing, but I don't want to be accused of shilling my Youtube channel or anything like that. It just seems like if a picture is worth 1000 words, a video is probably worth at least ten thousand.

 

As promised, I created a new thread for our next pile of cards, here:
https://hardforum.com/threads/graphics-card-necromancy-continued-radeon-r9-290x.1982565/

 

Um, the cap I measured was 11.8uF, not 120.

Without tearing the card down and looking, it's hard to say. If you're lucky, it's one of the zero ohm resistors that they seem to be using either as fuses, or to configure power distribution. I say get yourself the right torx bit and take the heatsink off and look for clues.

Either way, if you did this to two cards, undo it and get the right caps before you try it on the second one.

 

Sure. I can certainly take a look at it, anyway. Figure out how much it'll cost to ship, and PM me.

 

Where in Texas are you?

 

So like, you get a picture, and you can boot into Windows with the "VGA Display Adapter" driver, but when you try to install the real Nvidia/AMD driver, it doesn't recognize the card as compatible? Are you getting Error 43 in the device manager?

That could be indicative of a lot different things - a damaged or corrupted BIOS (hardware or software), some sort of GPU die or memory hardware failure, even problems unrelated to the card itself, such as the motherboard or just a corrupted Windows install. From memory, both AMD and Nvidia cards are capable of doing that, but they do it for different reasons, and each card design may be prone to different modes of failure.

That sort of problem is the hardest to troubleshoot without insider information from the card's manufacturer, but a corrupted BIOS is what I would suspect first (this could maybe be repaired without the need for soldering). After that, some sort of hardware failure within the GPU or memory would be next, either in the silicone itself, or the BGA solder balls holding it to the board. At that point, if all you want is a working card, just buying a working one is probably the best option, unless it's a super expensive 2080 Ti, or something. If that's the case, it should still be under warranty.

 

Oh, ok. I'm in the Austin area. Send me a PM with your paypal address.

 

Did you not read the thread? I got one working.

And you can get working ones for about $100-$120 on ebay. You could almost get two for the price of that SSD.