8600GTS Fails!

[Alpha736]

I have a Core i7 920 machine with 3gb of ram and an 8600 GTS. It's running Dual SMP MPICH clients with 4 threads each and a GPU2 client on the 8600 GTS.

The SMP clients are running smoothly but the GPU2 client is failing at roughly 25% on every workunit. It's giving UNSTABLE_MACHINE with a Corestatus of 7A, which seems like a generic GPU error.

Any ideas guys?

Here's the log, sorry it's kinda long, but I wanted to give the full log since the last restart...

--- Opening Log file [April 16 09:18:52 UTC]


# Windows GPU Console Edition #################################################
###############################################################################

Folding@Home Client Version 6.23

http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Users\Phylop\AppData\Roaming\Folding@home-gpu


[09:18:52] - Ask before connecting: No
[09:18:52] - User name: Alpha736 (Team 33)
[09:18:52] - User ID: 59399739685AB2F0
[09:18:52] - Machine ID: 3
[09:18:52]
[09:18:52] Loaded queue successfully.
[09:18:52] Initialization complete
[09:18:52] - Preparing to get new work unit...
[09:18:52] + Attempting to get work packet
[09:18:52] - Connecting to assignment server
[09:18:52] - Successful: assigned to (171.67.108.11).
[09:18:52] + News From Folding@Home: GPU folding beta
[09:18:53] Loaded queue successfully.
[09:18:54] + Closed connections
[09:18:54]
[09:18:54] + Processing work unit
[09:18:54] Core required: FahCore_11.exe
[09:18:54] Core found.
[09:18:54] Working on queue slot 09 [April 16 09:18:54 UTC]
[09:18:54] + Working ...
[09:18:54]
[09:18:54] *------------------------------*
[09:18:54] Folding@Home GPU Core - Beta
[09:18:54] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[09:18:54]
[09:18:54] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[09:18:54] Build host: amoeba
[09:18:54] Board Type: Nvidia
[09:18:54] Core :
[09:18:54] Preparing to commence simulation
[09:18:54] - Looking at optimizations...
[09:18:54] - Created dyn
[09:18:54] - Files status OK
[09:18:55] - Expanded 96680 -> 489240 (decompressed 506.0 percent)
[09:18:55] Called DecompressByteArray: compressed_data_size=96680 data_size=489240, decompressed_data_size=489240 diff=0
[09:18:55] - Digital signature verified
[09:18:55]
[09:18:55] Project: 5755 (Run 8, Clone 76, Gen 297)
[09:18:55]
[09:18:55] Assembly optimizations on if available.
[09:18:55] Entering M.D.
[09:19:01] Working on Protein
[09:19:04] Client config found, loading data.
[09:19:04] Starting GUI Server
[09:25:49] Completed 1%
[09:32:35] Completed 2%
[09:39:25] Completed 3%
[09:47:36] Completed 4%
[09:55:26] Completed 5%
[10:02:11] Completed 6%
[10:08:55] Completed 7%
[10:15:39] Completed 8%
[10:23:16] Completed 9%
[10:31:25] Completed 10%
[10:38:31] Completed 11%
[10:46:18] Completed 12%
[10:54:27] Completed 13%
[11:01:22] Completed 14%
[11:08:06] Completed 15%
[11:14:50] Completed 16%
[11:21:34] Completed 17%
[11:28:19] Completed 18%
[11:35:03] Completed 19%
[11:41:47] Completed 20%
[11:46:24] Run: exception thrown during GuardedRun
[11:46:24] Run: exception thrown in GuardedRun -- Gromacs cannot continue further.
[11:46:24] Going to send back what have done -- stepsTotalG=10000000
[11:46:24] Work fraction=0.2068 steps=10000000.
[11:46:28] logfile size=10720 infoLength=10720 edr=0 trr=23
[11:46:28] - Writing 11256 bytes of core data to disk...
[11:46:28] Done: 10744 -> 3931 (compressed to 36.5 percent)
[11:46:28] ... Done.
[11:46:28]
[11:46:28] Folding@home Core Shutdown: UNSTABLE_MACHINE
[11:46:32] CoreStatus = 7A (122)
[11:46:32] Sending work to server
[11:46:32] Project: 5755 (Run 8, Clone 76, Gen 297)
[11:46:32] - Read packet limit of 540015616... Set to 524286976.


[11:46:32] + Attempting to send results [April 16 11:46:32 UTC]
[11:46:33] + Results successfully sent
[11:46:33] Thank you for your contribution to Folding@Home.
[11:46:37] - Preparing to get new work unit...
[11:46:37] + Attempting to get work packet
[11:46:37] - Connecting to assignment server
[11:46:38] - Successful: assigned to (171.64.65.106).
[11:46:38] + News From Folding@Home: GPU folding beta
[11:46:38] Loaded queue successfully.
[11:46:39] + Closed connections
[11:46:44]
[11:46:44] + Processing work unit
[11:46:44] Core required: FahCore_11.exe
[11:46:44] Core found.
[11:46:44] Working on queue slot 00 [April 16 11:46:44 UTC]
[11:46:44] + Working ...
[11:46:44]
[11:46:44] *------------------------------*
[11:46:44] Folding@Home GPU Core - Beta
[11:46:44] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[11:46:44]
[11:46:44] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[11:46:44] Build host: amoeba
[11:46:44] Board Type: Nvidia
[11:46:44] Core :
[11:46:44] Preparing to commence simulation
[11:46:44] - Looking at optimizations...
[11:46:44] - Created dyn
[11:46:44] - Files status OK
[11:46:44] - Expanded 67939 -> 350980 (decompressed 516.6 percent)
[11:46:44] Called DecompressByteArray: compressed_data_size=67939 data_size=350980, decompressed_data_size=350980 diff=0
[11:46:44] - Digital signature verified
[11:46:44]
[11:46:44] Project: 5776 (Run 3, Clone 449, Gen 3)
[11:46:44]
[11:46:44] Assembly optimizations on if available.
[11:46:44] Entering M.D.
[11:46:51] mdrun_gpu returned
[11:46:51] Going to send back what have done -- stepsTotalG=0
[11:46:51] Work fraction=0.0000 steps=0.
[11:46:55] logfile size=0 infoLength=0 edr=0 trr=25
[11:46:55] - Writing 637 bytes of core data to disk...
[11:46:55] Done: 125 -> 123 (compressed to 98.4 percent)
[11:46:55] ... Done.
[11:46:55]
[11:46:55] Folding@home Core Shutdown: UNSTABLE_MACHINE
[11:46:59] CoreStatus = 7A (122)
[11:46:59] Sending work to server
[11:46:59] Project: 5776 (Run 3, Clone 449, Gen 3)
[11:46:59] - Read packet limit of 540015616... Set to 524286976.


[11:46:59] + Attempting to send results [April 16 11:46:59 UTC]
[11:46:59] + Results successfully sent
[11:46:59] Thank you for your contribution to Folding@Home.
[11:47:03] - Preparing to get new work unit...
[11:47:03] + Attempting to get work packet
[11:47:03] - Connecting to assignment server
[11:47:04] - Successful: assigned to (171.64.65.106).
[11:47:04] + News From Folding@Home: GPU folding beta
[11:47:04] Loaded queue successfully.
[11:47:05] + Closed connections
[11:47:10]
[11:47:10] + Processing work unit
[11:47:10] Core required: FahCore_11.exe
[11:47:10] Core found.
[11:47:10] Working on queue slot 01 [April 16 11:47:10 UTC]
[11:47:10] + Working ...
[11:47:10]
[11:47:10] *------------------------------*
[11:47:10] Folding@Home GPU Core - Beta
[11:47:10] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[11:47:10]
[11:47:10] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[11:47:10] Build host: amoeba
[11:47:10] Board Type: Nvidia
[11:47:10] Core :
[11:47:10] Preparing to commence simulation
[11:47:10] - Looking at optimizations...
[11:47:10] - Created dyn
[11:47:10] - Files status OK
[11:47:10] - Expanded 66299 -> 348500 (decompressed 525.6 percent)
[11:47:10] Called DecompressByteArray: compressed_data_size=66299 data_size=348500, decompressed_data_size=348500 diff=0
[11:47:10] - Digital signature verified
[11:47:10]
[11:47:10] Project: 5777 (Run 11, Clone 443, Gen 11)
[11:47:10]
[11:47:10] Assembly optimizations on if available.
[11:47:10] Entering M.D.
[11:47:17] mdrun_gpu returned
[11:47:17] Going to send back what have done -- stepsTotalG=0
[11:47:17] Work fraction=0.0000 steps=0.
[11:47:21] logfile size=0 infoLength=0 edr=0 trr=25
[11:47:21] - Writing 637 bytes of core data to disk...
[11:47:21] Done: 125 -> 124 (compressed to 99.2 percent)
[11:47:21] ... Done.
[11:47:21]
[11:47:21] Folding@home Core Shutdown: UNSTABLE_MACHINE
[11:47:25] CoreStatus = 7A (122)
[11:47:25] Sending work to server
[11:47:25] Project: 5777 (Run 11, Clone 443, Gen 11)
[11:47:25] - Read packet limit of 540015616... Set to 524286976.


[11:47:25] + Attempting to send results [April 16 11:47:25 UTC]
[11:47:25] + Results successfully sent
[11:47:25] Thank you for your contribution to Folding@Home.
[11:47:29] - Preparing to get new work unit...
[11:47:29] + Attempting to get work packet
[11:47:29] - Connecting to assignment server
[11:47:30] - Successful: assigned to (171.64.65.106).
[11:47:30] + News From Folding@Home: GPU folding beta
[11:47:30] Loaded queue successfully.
[11:47:31] + Closed connections
[11:47:36]
[11:47:36] + Processing work unit
[11:47:36] Core required: FahCore_11.exe
[11:47:36] Core found.
[11:47:36] Working on queue slot 02 [April 16 11:47:36 UTC]
[11:47:36] + Working ...
[11:47:36]
[11:47:36] *------------------------------*
[11:47:36] Folding@Home GPU Core - Beta
[11:47:36] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[11:47:36]
[11:47:36] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[11:47:36] Build host: amoeba
[11:47:36] Board Type: Nvidia
[11:47:36] Core :
[11:47:36] Preparing to commence simulation
[11:47:36] - Looking at optimizations...
[11:47:36] - Created dyn
[11:47:36] - Files status OK
[11:47:36] - Expanded 66268 -> 348500 (decompressed 525.8 percent)
[11:47:36] Called DecompressByteArray: compressed_data_size=66268 data_size=348500, decompressed_data_size=348500 diff=0
[11:47:36] - Digital signature verified
[11:47:36]
[11:47:36] Project: 5777 (Run 4, Clone 487, Gen 6)
[11:47:36]
[11:47:36] Assembly optimizations on if available.
[11:47:36] Entering M.D.
[11:47:43] mdrun_gpu returned
[11:47:43] Going to send back what have done -- stepsTotalG=0
[11:47:43] Work fraction=0.0000 steps=0.
[11:47:47] logfile size=0 infoLength=0 edr=0 trr=25
[11:47:47] - Writing 637 bytes of core data to disk...
[11:47:47] Done: 125 -> 124 (compressed to 99.2 percent)
[11:47:47] ... Done.
[11:47:47]
[11:47:47] Folding@home Core Shutdown: UNSTABLE_MACHINE
[11:47:50] CoreStatus = 7A (122)
[11:47:50] Sending work to server
[11:47:50] Project: 5777 (Run 4, Clone 487, Gen 6)
[11:47:50] - Read packet limit of 540015616... Set to 524286976.


[11:47:50] + Attempting to send results [April 16 11:47:50 UTC]
[11:47:51] + Results successfully sent
[11:47:51] Thank you for your contribution to Folding@Home.
[11:47:55] - Preparing to get new work unit...
[11:47:55] + Attempting to get work packet
[11:47:55] - Connecting to assignment server
[11:47:55] - Successful: assigned to (171.64.65.106).
[11:47:55] + News From Folding@Home: GPU folding beta
[11:47:55] Loaded queue successfully.
[11:47:57] + Closed connections
[11:48:02]
[11:48:02] + Processing work unit
[11:48:02] Core required: FahCore_11.exe
[11:48:02] Core found.
[11:48:02] Working on queue slot 03 [April 16 11:48:02 UTC]
[11:48:02] + Working ...
[11:48:02]
[11:48:02] *------------------------------*
[11:48:02] Folding@Home GPU Core - Beta
[11:48:02] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[11:48:02]
[11:48:02] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[11:48:02] Build host: amoeba
[11:48:02] Board Type: Nvidia
[11:48:02] Core :
[11:48:02] Preparing to commence simulation
[11:48:02] - Looking at optimizations...
[11:48:02] - Created dyn
[11:48:02] - Files status OK
[11:48:02] - Expanded 66345 -> 348500 (decompressed 525.2 percent)
[11:48:02] Called DecompressByteArray: compressed_data_size=66345 data_size=348500, decompressed_data_size=348500 diff=0
[11:48:02] - Digital signature verified
[11:48:02]
[11:48:02] Project: 5778 (Run 13, Clone 480, Gen 8)
[11:48:02]
[11:48:02] Assembly optimizations on if available.
[11:48:02] Entering M.D.
[11:48:08] mdrun_gpu returned
[11:48:08] Going to send back what have done -- stepsTotalG=0
[11:48:08] Work fraction=0.0000 steps=0.
[11:48:12] logfile size=0 infoLength=0 edr=0 trr=25
[11:48:12] - Writing 637 bytes of core data to disk...
[11:48:12] Done: 125 -> 124 (compressed to 99.2 percent)
[11:48:12] ... Done.
[11:48:12]
[11:48:12] Folding@home Core Shutdown: UNSTABLE_MACHINE
[11:48:16] CoreStatus = 7A (122)
[11:48:16] Sending work to server
[11:48:16] Project: 5778 (Run 13, Clone 480, Gen 8)
[11:48:16] - Read packet limit of 540015616... Set to 524286976.


[11:48:16] + Attempting to send results [April 16 11:48:16 UTC]
[11:48:16] + Results successfully sent
[11:48:16] Thank you for your contribution to Folding@Home.
[11:48:20] EUE limit exceeded. Pausing 24 hours.

 

[sirmonkey1985]

what temps are you hitting? and honestly if you ran 4 SMP clients.. 2 cores per client.. you would make triple what the 8600GTS makes..

its like getting the fastest race horse in the world.. and then putting the worlds fattest guy on it and telling the horse to run..

thats about the best way to explain the i7 vs having an 8600GTS running on it..

 

[Alpha736]

what temps are you hitting? and honestly if you ran 4 SMP clients.. 2 cores per client.. you would make triple what the 8600GTS makes..

its like getting the fastest race horse in the world.. and then putting the worlds fattest guy on it and telling the horse to run..

thats about the best way to explain the i7 vs having an 8600GTS running on it..

Yeah, but this isn't a gaming machine, it's a folding machine, so it shouldn't matter that the GPU is way slower than the CPU, it's not like a slow GPU is going to effect CPU performance...

The temps on the i7 are around 40-50C depending on ambient temps. The 8600GTS is getting up to about 60C. So the temps are fine, it must be something else causing problems with the GPU client.

 

[crazjayz]

Other than temps, try dropping one SMP and see if that works. Maybe check ram and cpu for stability? As you said, it's a seemingly random error...

 

[sirmonkey1985]

hmm then thats weird temps are fine.. the smp clients shouldnt be a problem because 2 smp clients arent really using 100% of all 8 threads even though it says it is.. the only thing i could think of is try downclocking the shaders a little bit and see if it EUE's again.. which drivers are you using and which OS?

 

[capreppy]

If it isn't temps, it is possibly drivers. I was on 178.24 (understood to be best for PpD), but I got a lot of EUEs. I switched to 182.08 and am rock solid stable and have been for weeks. I am also running 2 Linux SMP clients via VM. In addition, I am using WinAFC to make sure that the GPU client doesn't starve for CPU cycles.

 

[Alpha736]

Well, the drivers I am running on this machine are the latest ones which came out April 2nd, 182.50. Should I try rolling back to 182.08? I was never on any other set of drivers since I only set this machine up a few weeks ago...

Another thing I'm thinking it might be is the fact that this machine has multiple user accounts. I have a profile that I am constantly logged into so that I can run the GPU client. But it's my Dad's home office computer, and he has his own account that he logs into. Could the fact that the account that the GPU client is running on isn't logged into (EDIT: It is logged into, it's just running in the background, when my Dad is on his account), be causing this? I sorta don't think it is though, since it's consistently failing at 20%.

 

[Xilikon]

Yes, switching users can cause this issue and that's your answer. You need to stop the client, switch accounts then restart it so it can continue folding.

 

[Alpha736]

Yes, switching users can cause this issue and that's your answer. You need to stop the client, switch accounts then restart it so it can continue folding.

If that's what is causing the failures I might as well remove the GPU client unless I can find a workaround. There is no way I'm going to be able to micromanage that machine without annoying the hell out of my Dad.

 

[Imitation]

Yes, switching users can cause this issue and that's your answer. You need to stop the client, switch accounts then restart it so it can continue folding.

Can this also happen when i use mstsc into my machine? I use the same username as the one that's logged in at the time. I've been getting these about once a week.

 

[Alpha736]

Can this also happen when i use mstsc into my machine? I use the same username as the one that's logged in at the time. I've been getting these about once a week.

I can't say for sure, but it seems like it might, because RDP/mstsc replaces the display driver with it's own. I use logmein and I haven't had any problems with that...

 

[Xilikon]

Yes, that's correct as well... RDC/mstsc will also crash the GPU client since it replace the display driver as Alpha736 outlined. We use VNC or logmein instead since it doesn't mess with the client at all.