NANs detected on GPU issues while folding

[fx9]

I'm having some trouble with some WU's and I'm pretty sure its my machine as there are a lot of them that have failed, here are the snippets from my log file. The next unit after the last one has completed successfully so I'm confused. What is going on here? I have deleted the queue.dat and the work folder in the past since this isn't the first time, now the unstable machine message is happening more often. My 8800gt clocks are at 600/1783/900.

[07:22:18] Completed 45%
[07:23:05] Completed 46%
[07:23:05] mdrun_gpu returned
[07:23:05] NANs detected on GPU
[07:23:05]
[07:23:05] Folding@home Core Shutdown: UNSTABLE_MACHINE
[07:23:08] CoreStatus = 7A (122)
[07:23:08] Sending work to server
[07:23:08] Project: 5770 (Run 11, Clone 137, Gen 1448)
[07:23:08] - Error: Could not get length of results file work/wuresults_06.dat
[07:23:08] - Error: Could not read unit 06 file. Removing from queue.
[07:23:08] - Preparing to get new work unit...
[07:23:08] + Attempting to get work packet
[07:23:08] - Connecting to assignment server
[07:23:08] - Successful: assigned to (171.67.108.11).
[07:23:08] + News From Folding@Home: Welcome to Folding@Home
[07:23:08] Loaded queue successfully.
[07:23:09] + Closed connections
[07:23:14]
[07:23:14] + Processing work unit
[07:23:14] Core required: FahCore_11.exe
[07:23:14] Core found.
[07:23:14] Working on queue slot 07 [November 25 07:23:14 UTC]
[07:23:14] + Working ...
[07:23:16]
[07:23:16] *------------------------------*
[07:23:16] Folding@Home GPU Core - Beta
[07:23:16] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[07:23:16]
[07:23:16] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[07:23:16] Build host: amoeba
[07:23:16] Board Type: Nvidia
[07:23:16] Core :
[07:23:16] Preparing to commence simulation
[07:23:16] - Looking at optimizations...
[07:23:16] - Created dyn
[07:23:16] - Files status OK
[07:23:16] - Expanded 45415 -> 251112 (decompressed 552.9 percent)
[07:23:16] Called DecompressByteArray: compressed_data_size=45415 data_size=251112, decompressed_data_size=251112 diff=0
[07:23:16] - Digital signature verified
[07:23:16]
[07:23:16] Project: 5772 (Run 3, Clone 83, Gen 1113)
[07:23:16]
[07:23:16] Assembly optimizations on if available.
[07:23:16] Entering M.D.
[07:23:23] Working on Protein
[07:23:25] Client config found, loading data.
[07:23:25] Starting GUI Server
[07:24:20] Completed 1%
[07:25:15] Completed 2%
[07:26:09] Completed 3%
[07:27:03] Completed 4%
[07:27:58] Completed 5%
[07:28:52] Completed 6%
[07:29:47] Completed 7%
[07:30:41] Completed 8%
[07:31:36] Completed 9%
[07:32:30] Completed 10%
[07:33:25] Completed 11%
[07:34:19] Completed 12%
[07:35:14] Completed 13%
[07:36:08] Completed 14%
[07:37:03] Completed 15%
[07:37:57] Completed 16%
[07:38:52] Completed 17%
[07:39:46] Completed 18%
[07:40:41] Completed 19%
[07:41:36] Completed 20%
[07:42:30] Completed 21%
[07:43:24] Completed 22%
[07:44:19] Completed 23%
[07:45:13] Completed 24%
[07:46:08] Completed 25%
[07:47:02] Completed 26%
[07:47:57] Completed 27%
[07:48:51] Completed 28%
[07:49:45] Completed 29%
[07:50:40] Completed 30%
[07:51:34] Completed 31%
[07:52:29] Completed 32%
[07:53:24] Completed 33%
[07:54:18] Completed 34%
[07:55:13] Completed 35%
[07:56:07] Completed 36%
[07:57:02] Completed 37%
[07:57:56] Completed 38%
[07:58:48] Completed 39%
[07:58:48] mdrun_gpu returned
[07:58:48] NANs detected on GPU
[07:58:48]
[07:58:48] Folding@home Core Shutdown: UNSTABLE_MACHINE
[07:58:52] CoreStatus = 7A (122)
[07:58:52] Sending work to server
[07:58:52] Project: 5772 (Run 3, Clone 83, Gen 1113)
[07:58:52] - Error: Could not get length of results file work/wuresults_07.dat
[07:58:52] - Error: Could not read unit 07 file. Removing from queue.
[07:58:52] - Preparing to get new work unit...
[07:58:52] + Attempting to get work packet
[07:58:52] - Connecting to assignment server
[07:58:53] - Successful: assigned to (171.67.108.11).
[07:58:53] + News From Folding@Home: Welcome to Folding@Home
[07:58:53] Loaded queue successfully.
[07:58:54] + Closed connections
[07:58:59]
[07:58:59] + Processing work unit
[07:58:59] Core required: FahCore_11.exe
[07:58:59] Core found.
[07:58:59] Working on queue slot 08 [November 25 07:58:59 UTC]
[07:58:59] + Working ...
[07:59:00]
[07:59:00] *------------------------------*
[07:59:00] Folding@Home GPU Core - Beta
[07:59:00] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[07:59:00]
[07:59:00] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[07:59:00] Build host: amoeba
[07:59:00] Board Type: Nvidia
[07:59:00] Core :
[07:59:00] Preparing to commence simulation
[07:59:00] - Looking at optimizations...
[07:59:01] - Created dyn
[07:59:01] - Files status OK
[07:59:01] - Expanded 45341 -> 251112 (decompressed 553.8 percent)
[07:59:01] Called DecompressByteArray: compressed_data_size=45341 data_size=251112, decompressed_data_size=251112 diff=0
[07:59:01] - Digital signature verified
[07:59:01]
[07:59:01] Project: 5772 (Run 6, Clone 260, Gen 759)
[07:59:01]
[07:59:01] Assembly optimizations on if available.
[07:59:01] Entering M.D.
[07:59:08] Working on Protein
[07:59:09] Client config found, loading data.
[07:59:09] Starting GUI Server
[08:00:03] Completed 1%
[08:00:57] Completed 2%
[08:01:52] Completed 3%
[08:02:47] Completed 4%
[08:03:41] Completed 5%
[08:04:35] Completed 6%
[08:05:30] Completed 7%
[08:06:24] Completed 8%
[08:07:19] Completed 9%
[08:08:13] Completed 10%
[08:09:08] Completed 11%
[08:10:03] Completed 12%
[08:10:57] Completed 13%
[08:11:51] Completed 14%
[08:12:46] Completed 15%
[08:13:40] Completed 16%
[08:14:35] Completed 17%
[08:15:29] Completed 18%
[08:16:24] Completed 19%
[08:17:18] Completed 20%
[08:18:13] Completed 21%
[08:19:07] Completed 22%
[08:20:02] Completed 23%
[08:20:56] Completed 24%
[08:21:51] Completed 25%
[08:22:45] Completed 26%
[08:23:40] Completed 27%
[08:24:34] Completed 28%
[08:25:29] Completed 29%
[08:26:23] Completed 30%
[08:27:18] Completed 31%
[08:28:12] Completed 32%
[08:29:07] Completed 33%
[08:30:02] Completed 34%
[08:30:56] Completed 35%
[08:31:50] Completed 36%
[08:32:45] Completed 37%
[08:33:39] Completed 38%
[08:34:34] Completed 39%
[08:35:28] Completed 40%
[08:36:23] Completed 41%
[08:37:17] Completed 42%
[08:38:12] Completed 43%
[08:39:07] Completed 44%
[08:40:01] Completed 45%
[08:40:56] Completed 46%
[08:41:50] Completed 47%
[08:42:45] Completed 48%
[08:43:39] Completed 49%
[08:44:34] Completed 50%
[08:45:28] Completed 51%
[08:46:23] Completed 52%
[08:47:17] Completed 53%
[08:47:17] mdrun_gpu returned
[08:47:17] NANs detected on GPU
[08:47:17]
[08:47:17] Folding@home Core Shutdown: UNSTABLE_MACHINE
[08:47:20] CoreStatus = 7A (122)
[08:47:20] Sending work to server
[08:47:20] Project: 5772 (Run 6, Clone 260, Gen 759)
[08:47:20] - Error: Could not get length of results file work/wuresults_08.dat
[08:47:20] - Error: Could not read unit 08 file. Removing from queue.
[08:47:20] - Preparing to get new work unit...
[08:47:20] + Attempting to get work packet
[08:47:20] - Connecting to assignment server
[08:47:20] - Successful: assigned to (171.64.65.20).
[08:47:20] + News From Folding@Home: Welcome to Folding@Home
[08:47:21] Loaded queue successfully.
[08:47:22] + Closed connections
[08:47:27]
[08:47:27] + Processing work unit
[08:47:27] Core required: FahCore_14.exe
[08:47:27] Core found.
[08:47:27] Working on queue slot 09 [November 25 08:47:27 UTC]
[08:47:27] + Working ...
[08:47:28]
[08:47:28] *------------------------------*
[08:47:28] Folding@Home GPU Core - Beta
[08:47:28] Version 1.26 (Wed Oct 14 13:09:26 PDT 2009)
[08:47:28]
[08:47:28] Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[08:47:28] Build host: vspm46
[08:47:28] Board Type: Nvidia
[08:47:28] Core :
[08:47:28] Preparing to commence simulation
[08:47:28] - Looking at optimizations...
[08:47:28] - Created dyn
[08:47:28] - Files status OK
[08:47:28] - Expanded 70307 -> 360060 (decompressed 512.1 percent)
[08:47:28] Called DecompressByteArray: compressed_data_size=70307 data_size=360060, decompressed_data_size=360060 diff=0
[08:47:28] - Digital signature verified
[08:47:28]
[08:47:28] Project: 5910 (Run 9, Clone 49, Gen 15)
[08:47:28]
[08:47:29] Assembly optimizations on if available.
[08:47:29] Entering M.D.
[08:47:35] Tpr hash work/wudata_09.tpr: 2778623592 487559074 3292183646 224567047 3511238789
[08:47:35] Working on Protein
[08:47:36] Client config found, loading data.
[08:47:37] Starting GUI Server

 

[Nessus85]

Those are some pretty high shader clocks are they not? Maybe the card is too far overclocked?

 

[fx9]

I'm pretty sure its not temperatures, as its 68C while folding. Its been working for a couple weeks without a hitch, then all of a sudden a surge of bad WUs which none were reported as bad. I'm pretty sure its the card. I'll try setting it back to default clocks when I get home tonight.

 

[Vaulter98c]

yea, thats a crazy OC, Id point my finger at that for now

EDIT
curious tho, what were you seeing PPD from that OC? Ive got 4 8800's that I cant seem to get above 1600 ish, wouldnt mind doing a voltmod if I felt it was worth it tho

 

[Wesleynator]

This happens from time to time. You'll think it is a problem with a particular card or machine and then it will go away just like it appeared. I run 2x880gt and 3x9800gt plus one gtx260/216 and it will randomly happen occasionally to any one of them. My 8800gt's all run at 1782 shaders no problem normally but when this happens I will back the card down a step or two (i.e. 1728, 1674) which seems to help then after a day or two creep it back up. It doesn't happen very often and my theory is it is a batch of sensitive wu's. I see yours is a 5772 which, if I'm not mistaken, is a 353 pointer and it seems to happen a bit more often with those.

 

[fx9]

yea, thats a crazy OC, Id point my finger at that for now

EDIT
curious tho, what were you seeing PPD from that OC? Ive got 4 8800's that I cant seem to get above 1600 ish, wouldnt mind doing a voltmod if I felt it was worth it tho

The 2nd entry on my Fahmon screenshot should answer your question. Its worked for a couple weeks without any errors though which is why i'm confused its happening now.

 

[Vaulter98c]

hmm
might have to build that 4 way today after all lol

 

[Elledan]

OCs are known to degrade after a while. As I said in the IRC channel, lower the GPU clocks and see whether that does anything. I'm not sure there are any other benchmarks beyond F@H to test computational stability of a GPU, else I would recommend using one