SR-2 Optimization Thread

[FLECOM]

so my new SR-2 rig (Dual L5640 @ 3.6ghz) is netting me 55kppd on a P6075... does that sound right?

 

[DooKey]

That's about right. Those standard SMP wu really cut down the PPD compared to bigadv.

 

[capreppy]

so my new SR-2 rig (Dual L5640 @ 3.6ghz) is netting me 55kppd on a P6075... does that sound right?

That is actually 20k PpD low. I am getting 75k PpD on 6071/6072 (pretty consistent on anything other than a P6701). I am also @ 3.6 w/dual L5640's.

Anything else running on that box? Do you have NUMA turned off?
What are your TPFs? I'm at 1:09 for normal A3 WUs.
What RAM are you using?

 

[DooKey]

That is actually 20k PpD low. I am getting 75k PpD on 6071/6072 (pretty consistent on anything other than a P6701). I am also @ 3.6 w/dual L5640's.

Anything else running on that box? Do you have NUMA turned off?
What are your TPFs? I'm at 1:09 for normal A3 WUs.
What RAM are you using?

My bad, you said 6071 not 6701.......

 

[capreppy]

My bad, you said 6071 not 6701.......

Love that ninja edit, but I got the notice before the edit

 

[FLECOM]

nothing is running on the box... NUMA off... 3x4gb silverline kit from mushkin... I have a second 3x4gb silverline kit but when I install it Windows 7 x64 blue screens with the following error

the driver is mismanaging system PTEs

 

[capreppy]

nothing is running on the box... NUMA off... 3x4gb silverline kit from mushkin... I have a second 3x4gb silverline kit but when I install it Windows 7 x64 blue screens with the following error

With NUMA off and only one bank of memory populated, should that even work? Obviously it does, but I think that may be why your PpD is off.

Why 3x4GB? This is obviously not a dedicated folder and you'll be doing other stuff on this box.

edit: You may need more DIMM voltage to use that second 3x4GB kit. just a thought

 

[FLECOM]

DIMM voltage is set to 1.65... these are marked 1.5V

its going to be my workstation just haven't gotten everything put together for it so right now its a dedicated folder...

had a 3x4gb kit in my i7 920 desktop which now is a dedicated folder with 3x1gb in it and wanted to do 6 channel so I just got a second 3x4gb kit instead of buying 6x2gb or something like that

the timing kinda sucks tho 9-9-9-24

also I just refreshed HFM, machine is now down to 49500ppd

 

[capreppy]

Do you have an extra set of 6x RAM? If so, see if changing out the RAM (to exclude the RAM as the issue). You've obviously bumped up the voltage so that definitely isn't the issue. I do wonder however if it is too much voltage? Since you'll be running these in 1T and you're only populating 6 slots vs 12, if you did indeed run it at 1.5 (or just over it) if your memory issues would go away.

The timings will have an impact, but should be neglible.

49k PpD: something is definitely wrong. That is what I get with 6701/2's.

 

[FLECOM]

I don't have any extra dimms that dont suck

I am going to try lowering the voltage and putting the other set of 3 to see what happens

 

[FLECOM]

ok so I lowered the dimm voltage to 1.5V again and put in the other 3... at first it was still seeing 12gb of ram... then I set everything that had to do with the memory to auto except for the vdimm... now it sees 20gb...

I have never had so many problems with a motherboard, I am ready to throw this piece of $hit out the window

edit: ok according to CPU-Z all 24gb are there, but it set my clocks to 11-11-11-29!!! AHHH!!!!

edit again: and now it blue screened with an IRQ NOT LESS OR EQUAL...

*bashes head into desk*

edit again (why not?): ok so I set the timings to 9-9-9-24 and the bios and windows only see 12gb but CPU-Z shows 24gb...

 

[capreppy]

Try the following:
Set 9-9-9-24 manually in BIOS. Leave everything else Auto. Do check to ensure that it is running 1T
Set DIMM Voltage to 1.5 (already there based on what you wrote above, but just checking)

CPU Frequency - 200
PCIE Frequency - 101
CPU Multiplier - 18
QPI Frequency - 4.800GT
Memory Frequency - DDR-1066
MCH Strap - Auto
CPU Uncore = 18x

These are the settings I am using (and most others) to get DDR3-1600.

 

[SazanEyes]

My first board had a bad DIMM slot and I had to RMA it. It sounds like your board may be bad too. If RAM is not being detected by BIOS, it's usually a hardware issue and nothing to do with overclocking. (Of course, if everything works without any overclock at all, then never mind.)

Can you try RAM only in the second CPU's bank (assuming that's the one giving you trouble) and disabling the first CPU with the jumper on the mobo? You need to try to isolate the problem. You may also need to try reseating the CPUs, or swapping the two CPUs between sockets. EVGA will ask you to try the same stuff before you RMA.

 

[404 - File not found]

My first board had a bad DIMM slot and I had to RMA it. It sounds like your board may be bad too. If RAM is not being detected by BIOS, it's usually a hardware issue and nothing to do with overclocking. (Of course, if everything works without any overclock at all, then never mind.)

Can you try RAM only in the second CPU's bank (assuming that's the one giving you trouble) and disabling the first CPU with the jumper on the mobo? You need to try to isolate the problem. You may also need to try reseating the CPUs, or swapping the two CPUs between sockets. EVGA will ask you to try the same stuff before you RMA.

My board is in RMA for the same reason, bad cpu1_dimm3 slot. I'm really disappointed they didn't test this...
Yeah isolate each side find out if theres a stick not booting, switch ram around see it's not a ram stick that's weaker, then switch cpu around (really less likely this is the problem).
For me, cpuz did report all the ram but bios/windows did not.

 

[10e]

@10e - Canadian beer? Thanks, I think... I am sure you mean well.. ;P

We have good beer! It's good stuff! I'm telling you! You would want it south of the equator!

ok so I lowered the dimm voltage to 1.5V again and put in the other 3... at first it was still seeing 12gb of ram... then I set everything that had to do with the memory to auto except for the vdimm... now it sees 20gb...

I have never had so many problems with a motherboard, I am ready to throw this piece of $hit out the window

edit: ok according to CPU-Z all 24gb are there, but it set my clocks to 11-11-11-29!!! AHHH!!!!

edit again: and now it blue screened with an IRQ NOT LESS OR EQUAL...

*bashes head into desk*

edit again (why not?): ok so I set the timings to 9-9-9-24 and the bios and windows only see 12gb but CPU-Z shows 24gb...

FLECOM, why not try stock, and go from there? This should help you isolate bad RAM or a bad DIMM slot... Also some people have had success re-seating their CPUs due to uneven socket pressure. That might be something you wanna try prior to RMA'ing the board.

 

[FLECOM]

Try the following:
Set 9-9-9-24 manually in BIOS. Leave everything else Auto. Do check to ensure that it is running 1T
Set DIMM Voltage to 1.5 (already there based on what you wrote above, but just checking)

CPU Frequency - 200
PCIE Frequency - 101
CPU Multiplier - 18
QPI Frequency - 4.800GT
Memory Frequency - DDR-1066
MCH Strap - Auto
CPU Uncore = 18x

These are the settings I am using (and most others) to get DDR3-1600.

thats exactly how my settings were, did not work... only saw 12gb of ram and unstable... if I set it to those settings except set the ram to DDR-800 it sees all 24gb and works fine

We have good beer! It's good stuff! I'm telling you! You would want it south of the equator!



FLECOM, why not try stock, and go from there? This should help you isolate bad RAM or a bad DIMM slot... Also some people have had success re-seating their CPUs due to uneven socket pressure. That might be something you wanna try prior to RMA'ing the board.

the machine worked fine before I tried to install this additional ram, so I don't think the CPU's would be an issue??

 

[may i be worthy]

try MCH strap of 1600, then maybe 1333...

 

[may i be worthy]

Ok, there are always lots of questions about what is the best case for the SR2... after wasting a lot of money on the Lian Li PC-P80B, I have found a better solution.... drum roll...

It is pricey, but worth it. $4.50 from Bunnings Warehouse. What's more, it is clearly made for folding.... CLOTHES! .....SEE WHAT I DID THERE?!

And here it is - just the right size for flopping my PSU and HDD over on top like half a pancake. I might add some more wire to stop curious 2 year old fingers going into a fan..

Why do I need to to bunch up and make room? For the new kid of course? Meet the latest edition, whose CPUs have been promised to be delivered in the next hour. (not holding my my breath, considering the "Guaranteed" delivery date was 5 days ago)

All three, together at last. Moar D

EDIT: Here in Oz we are still getting the version of the SR2 with the ECP panel, phew!

 

[FLECOM]

try MCH strap of 1600, then maybe 1333...

I will try that tomorrow...

currently got a 6701 and getting 40kppd

 

[jebo_4jc]

Good stuff mibw. I'm still leaning toward a $140 700d. A couple mods and it should be great.
Posted via [H] Mobile Device

 

[FLECOM]

I put mine in a rocketfish case, no mods necessary really... you just don't get all the PCIe slots

 

[jebo_4jc]

The rocket fish case was probably the best option. Too bad they're all gone.
Posted via [H] Mobile Device

 

[capreppy]

Ok. So after nearly 2 weeks of rock solid stability, I am getting BSODs.

Stop code 50. After looking around on the web, it looks like some sort of card reader error, but I don't have a card reader. The suggestions on the web were to do a re-install of the OS (Win7 x64 in my case).

I just completed a re-install of the OS and am still BSOD'ing

Thoughts / Ideas?

 

[may i be worthy]

I think that one is memory related.. run HCI memtest...

 

[capreppy]

I think that one is memory related.. run HCI memtest...

Running now. How many passess?

 

[capreppy]

Running now. How many passess?

Pass 1 complete and no errors

 

[capreppy]

My BSODs are alternating between 50 and D1. Memtest is fine.

Have re-installed the OS already. At my wits end on how to diagnose this issue.


11/12/10: 2050: Well it looks as if bumping up the VCore a teeny bit & bumping VDIMM one notch seems to have corrected my stability issue.

 

[[H]ecklerKoc[H]]

Currently working a 2684 . Here is what I have got.

Project ID: 2684
Core: GRO-A3
Credit: 8955
Frames: 100

Name: cpu smp-24
Path: C:\Program Files (x86)\FAH\smp\
Number of Frames Observed: 300

Min. Time / Frame : 00:16:29 - 92,027.9 PPD
Avg. Time / Frame : 00:20:08 - 68,173.2 PPD
Cur. Time / Frame : 00:18:12 - 82,033.3 PPD
R3F. Time / Frame : 00:17:23 - 85,887.2 PPD
All  Time / Frame : 00:16:56 - 88,169.6 PPD
Eff. Time / Frame : 00:17:10 - 86,971.2 PPD

HFM is showing 16:56 at this very second.

CPUs Installed: X5680, Costa Rica, "A" Batch.
HSF: Corsair H70
Memory Installed: G.Skill F3-16000CL6T-6GBPID (2GBx3), 2 kits, 6 DIMMs, 12GB total
BIOS: A50
NUMA: Disabled
Signal Tweak IOH QPI 0 Signal: -86
Signal Tweak IOH QPI 1 Signal: -16

Current Uptime: 4d 23:58:57

Current Info:
(as read from E-LEET version 1.08.6)

CPU Clock: 4256 MHz
CPU Multiplier: 23.0x
CPU FSB: 185 MHz
Memory Clock: 925
memory Speed: DDR3-1850 MHZ
DRAM FSB Ratio: 2:10
Memory Timings: 6-9-6-29-88-CR1

CPU0 VCore: 1.38 V - (Actual Setting: 1.375 V)
CPU0 VTT: 1.41 V - (Actual Setting: 1.375 V)
CPU0 DRAM: 1.68 V - (Actual Setting: 1.65 V)

CPU1 VCore: 1.35 V - (Actual Setting: 1.375 V)
CPU1 VTT: 1.37 V - (Actual Setting: 1.375 V)
CPU1 DRAM: 1.68 V - (Actual Setting: 1.65 V)

IOH VCore: 1.40 V - (Actual Setting: 1.4 V)
IOH PLL: 1.80 V - (Actual Setting: 1.8 V)

CPU #1 / Core #1: 83 C
CPU #1 / Core #2: 81 C
CPU #1 / Core #3: 81 C
CPU #1 / Core #4: 79 C
CPU #1 / Core #5: 84 C
CPU #1 / Core #6: 81 C
CPU #2 / Core #1: 88 C
CPU #2 / Core #2: 90 C
CPU #2 / Core #3: 86 C
CPU #2 / Core #4: 84 C
CPU #2 / Core #5: 90 C
CPU #2 / Core #6: 90 C
Room Ambient: 79 F (26.1 C)​

CPU-Z Validation ID: 1479735

Update:
Just bumped the settings a bit. Hopefully these new tryouts also stay up another four days with better TPFs.

CPU Clock: 4302 MHz
CPU Multiplier: 23.0x
CPU FSB: 187 MHz
Memory Clock: 935
memory Speed: DDR3-1870 MHZ
DRAM FSB Ratio: 2:10
Memory Timings: 6-9-6-24-74-CR1
Memory Frequency: DDR-1333 MHz
MCH Strap: 1333 MHz​

And of course something happened. A little BSOD, so trying a little more VCore.

CPU0 VCore: 1.38 V - (Actual Setting: 1.38125 V) (A little BSOD, so a little more VCore)
CPU1 VCore: 1.36 V - (Actual Setting: 1.38125 V) (A little BSOD, so a little more VCore)

CPU #1 / Core #1: 88 C
CPU #1 / Core #2: 86 C
CPU #1 / Core #3: 83 C
CPU #1 / Core #4: 82 C
CPU #1 / Core #5: 85 C
CPU #1 / Core #6: 91 C
CPU #2 / Core #1: 90 C
CPU #2 / Core #2: 92 C
CPU #2 / Core #3: 86 C
CPU #2 / Core #4: 84 C
CPU #2 / Core #5: 90 C
CPU #2 / Core #6: 91 C
Room Ambient: 79 F (26.1 C)​

 

[jebo_4jc]

thats nice

 

[404 - File not found]

hehe that's epic, HK.

I'm currently working on my overclock. 3564mhz stable.
198x 18
ram 6-9-6-28-1T
DDR-1066 (2:8)
I'm using zero2dash's template for the rest.

A47
17x (18x w/turbo) x 200
PCIe 101
QPI 4.800GT (18x, runs at 3600)
Uncore 2133 (16x)
Without Vdroop
Vcc boot/eventual 1.20
Vtt boot/eventual 1.30
DIMM 1.65
CPU PLL 1.65
IOH 1.4
IOH PLL 1.8
ICH I/O 1.5
ICH 1.125
IOH QPI 0 Signal -85
IOH QPI 1 Signal -16

Booting at 204blck 3.67ghz right now but not stable in linx.

 

[nimbyfaygo]

hehe that's epic, HK.

I'm currently working on my overclock. 3564mhz stable.
198x 18
ram 6-9-6-28-1T
DDR-1066 (2:8)
I'm using zero2dash's template for the rest.


Booting at 204blck 3.67ghz right now but not stable in linx.

I've not had any luck getting stable at 200 or over. I've been stable folding for about 3 days with 198bclk.
If you get it running past 200 I want to know how you did it

 

[zero2dash]

Ok, memory timings. This is voodoo shit, so if you are lost follow the recipe: First worry about divider speed. Do 1333 / 2:10 if you can. Then worry about if you can keep 1T command rate. Lastly lower your timings.

Just wanted to slight update....I'm working on disabling NUMA and getting rid of B2B.
In my case changing the MCH Strap to 1600 solved my disappearing ram problem. MCH Strap @ 1333, I lose 2gb (down to 10gb). If I change MCH Strap to 1600, I boot with all 12.
Initially I had MCH Strap on Auto, ram would disappear....that's why I tried B2B. Now I find with B2B Auto/0 (disabled), MCH Strap to 1600....I lose no ram.

Will update more soon....

 

[may i be worthy]

[H]k - something is still wrong. I can't work out what - you have all the right settings, but you are doing about a minute worse than either of my machines - you should be right in between them at 4.25. Even a tad faster clock for clock because your ram timings are faster. So at 4.3 you should be the fastest rig in the horde.

Of course the first thing to do is find time to run 2-4 frames of my 2686 bench, so we can rule out variation in 2684s.

The only thing that occurs is your temps are horrific - so close to the 95c limit (is that the case on x5680s?), you might be getting CPU throttling? My temps are very high 70's at 25c ambient.

Anyway, this is driving me batty - lets nut it out.

repost of my times:

From HFM: P2684

Name: SR2-1 - X5650 - 4200mhz
Number of Frames Observed: 300

Min. Time / Frame : 00:15:34 - 100,275 PPD
Avg. Time / Frame : 00:16:08 - 95,039 PPD


Name: SR2-2 - X5660 - 4300mhz
Number of Frames Observed: 300

Min. Time / Frame : 00:15:21 - 102,406 PPD
Avg. Time / Frame : 00:15:42 - 99,001 PPD

 

[may i be worthy]

Just wanted to slight update....I'm working on disabling NUMA and getting rid of B2B.
In my case changing the MCH Strap to 1600 solved my disappearing ram problem. MCH Strap @ 1333, I lose 2gb (down to 10gb). If I change MCH Strap to 1600, I boot with all 12.
Initially I had MCH Strap on Auto, ram would disappear....that's why I tried B2B. Now I find with B2B Auto/0 (disabled), MCH Strap to 1600....I lose no ram.

Will update more soon....

Awesome news! I recommended this to flecom, but that was just my theory, great to see it actually working.

 

[zero2dash]

I'm going to fart around more tomorrow with it. Going to wait til this 2685 finishes first though, don't feel like eating a point loss 'cause I sat on it for awhile goofing around in the bios.
Definitely seems like I really need to run it at 1600 on the strap.
Noticed digging around for OC info that safield ran his 2000 ram at 1600 strap and also 2T....changing to 2T may also solve some of my other problems (hopefully disabling NUMA).
It does look like with MCH Strap on auto, it is actually using strap 1600 timings (the what I would call "extended" timings). I put in 9-9-9-24 per the ram, the rest on auto pulls up
(full timings) 9-9-9-24-12-88-6-6-24-1T
I tried 1333, that gets 9-9-9-24-10-74-5-5-5-20-1T

I'm hoping once I get this ram setup flawless that disabling NUMA takes on the 1st try and stays with a reboot. Right now I can disable NUMA, boot, but when I reboot, I get the stupid fail to boot cycle (0C -> 2A, reboot).

This is the exact ram I'm running. @2000, 9-9-9-24 1.65vdimm
If I were to have to loosen timings, what should I go to...10-10-10-27?

 

[sfield]

I still have B2B CAS delay set to 0. I found that setting the Channel A/B/C latency values (aka RTL - round trip latency) eliminated cases of missing memory (NUMA enabled). The IMC initialization code was occasionally picking too low values for A/B/C when set to auto, which did occasionally lead to one module being "taken out". Your mileage may vary here, but worth checking at any rate.

I still can't get NUMA disabled working properly -- machine doesn't post. This is even with fiddling with B2B CAS delay. I have theories about why this is, but haven't had time to fully investigate..

There's a formula and spreadsheet for calculating appropriate RTL values here:

http://i4memory.com/f83/questions-about-round-trip-latency-24362/


I'm running the 6x4GB gskill kit at different timings now, and need to update the post you referenced:
7-8-7-24-128-1T A/B/C = 53/55/57, DDR3-1850.

185x23 w/x5680s.

I was running the same kit at DDR3-2050 8-9-8-24-2T when I was using x5650s in the machine. Early on, I tried 1T, which was unreliable, but that was before I nailed all the other settings down. now 1T is reliable (and within spec per gskill).



HK -- those temperatures are really high. limit on x5680 is 101C. My watercooled SR-2 processors at similiar settings to your setup is running 50-55C under max load.

You may want to tweak the vcore for each CPU per the thread below. I'm interested to hear what voltages you settle on. you could also push VTT down by adjusting the uncore multiplier down. I've tested stable as low as 1.275-1.3vvtt at 18x. 1.325vtt at 19x uncore.
http://www.evga.com/forums/tm.aspx?m=651731

I'm going to fart around more tomorrow with it. Going to wait til this 2685 finishes first though, don't feel like eating a point loss 'cause I sat on it for awhile goofing around in the bios.
Definitely seems like I really need to run it at 1600 on the strap.
Noticed digging around for OC info that safield ran his 2000 ram at 1600 strap and also 2T....changing to 2T may also solve some of my other problems (hopefully disabling NUMA).
It does look like with MCH Strap on auto, it is actually using strap 1600 timings (the what I would call "extended" timings). I put in 9-9-9-24 per the ram, the rest on auto pulls up
(full timings) 9-9-9-24-12-88-6-6-24-1T
I tried 1333, that gets 9-9-9-24-10-74-5-5-5-20-1T

I'm hoping once I get this ram setup flawless that disabling NUMA takes on the 1st try and stays with a reboot. Right now I can disable NUMA, boot, but when I reboot, I get the stupid fail to boot cycle (0C -> 2A, reboot).

This is the exact ram I'm running. @2000, 9-9-9-24 1.65vdimm
If I were to have to loosen timings, what should I go to...10-10-10-27?

 

[may i be worthy]

@ safield Thanks for the formula link, (a bit above my head. I get a bit lost with the theory, but I do know to just keep trying and testing until I get something I can have confidence in)

I found better results very early on setting memory settings manually (all except B2B). But I also get the missing memory with all settings set to the same settings as they go to Auto, just manually set- they had to be loosened a certain amount to get 100% reliable bootups. (with qualifier that one rig had 12dimms, and one rig 2:10 at 1870ddr using 1600ddr mem - so both had reason to need loosening beyond spec settings, which becomes a bit academic) Are you saying the RTL or A/B/C are the likely culprit?

I have not had time or the patience to do a thousand reboots to try reducing some of the settings to see which ones were the ones doing the damage at boot. Could be a lot of work for little benefit. But once I stop buildng one SR2 a month I will probably get restless and want to try again.

@zero2dash - when trying looser settings, loosen all of them (apart from 1T and B2B in proportion - it may not be optimal, but it is proven to work. Also beware you can get picky combos that will not boot at all. On mine 7 8 7 20 or 8 9 8 24 will boot, but 8 8 8 24 which is in between will not post - could be a kingston thing, but the reason I mention it is so you try a few combos. Given that you are at 9 9 9 24 already, I might suggest going to 2T and try that - which will probably work very well, and then try 8 8 8 20 to make up for the big hit you take going to 2T. (and if you are super lucky, maybe even 7 7 7 20)

 

[SazanEyes]

I finally swapped my X5650s between sockets, and now my temps are almost identical on each processor (unlike the 10 degree difference before). I have the "hot" CPU in the "cold" socket, and vice versa. I also rotated the heatsinks so that they're blowing out to the side rather than CPU0 blowing on CPU1, but I might change that later. Now to get back to overclocking...

 

[FLECOM]

does anyone here use their SR-2 rig as their workstation?

I was thinking about it today, I was planning on using my SR-2 as my main workstation (mostly browsing the internet, some TF2...) or would that be a bad idea?

 

[SazanEyes]

Mine will be my primary PC. As soon as I finish overclocking tests this weekend, I'm going to start installing software and copying data from my old desktop. Currently trying to find max stable BCLK. I keep getting BSODs when running LinX @ 20GB, so I'm reducing BCLK by 2MHz each time -- down to 184 MHz now. My other SR-2 is going to be a Hyper-V server. Technically I don't have any dedicated folding boxen, they just tend to spend a lot of time folding.

Based on times when I've shut down for an hour or two to tweak the BIOS, you don't lose many points per WU, so it shouldn't really matter. If you game for several hours a day, that might make a greater impact. MIBW apparently uses his for graphics rendering, which kills his PPD since he's not folding even close to 24 hours a day.