SR-2 Optimization Thread

Hi MIBW,

Move the controller over to another computer, took a little while swapping cards around and getting everything to show up in the right places. Its up and running and I can see all my data, I guess one of the good things about separate controllers vs onboard, is that you can move them about.

So back to SR-2, without the controller its booting as it should for the time being, but i thought ok, what is causing this, and I installed a second raid controller, same as the other one, and it booted up just fine. add all the ram back in and she is still fine.

Not sure what is going on here, I think its all running at default, not actually sure. I will go under the hood after this and check.

Got to finish this montage, adding and moving people around in a photo - nothing but joy :)

fingers crossed!

thanks - i'll b back!
 
Hello again to the crew (MIBW, musky, 10e, etc) :D

Looking for some guidance on an OC'd pair of x5670s, but most of the templates here seem to be x5650 or x5660. Even with a decent OC, I'm still not seeing the same TFP (can only reach ~14m, and this is under linux).

Any-who, also wanted to know what kind of ppd increase you've seen with the 'new' linux core on the SR-2, if you've tried it?

Always a fan,
-JT
 
may i be worthy said:
Hey jmtyra,

I have all my settings here: http://hardforum.com/showpost.php?p=1036223866&postcount=142
Also a link to x5680 here http://www.pbase.com/davexl/x5680
Click to expand...
Sweet, I'll check those out -- not sure how I missed that before. :eek:

may i be worthy said:
That sounds slow for a 2686/2692/6900/6901 but ballpark for a 2684. What settings are you at now?

Linux speedup post and thread here: http://hardforum.com/showpost.php?p=1036996899&postcount=411
Click to expand...
/agreed, it does seem slow. I'm running a few more benchmarks to be sure. Just finished a 2686, HFM info below. Still on A47 for stability, using these settings:
191 x 21 (no SS) = 4014, 1333ddr, MCH @ D ratio, qpi @ auto/4.8
volts: both cpus @ vcore 1.34375, vtt @ 1.375, ram @ 1.65 (9-9-9-26, 1t)

Thanks for the link on the Linux speedup, I'll check that out. Also I'll look through your template/settings and double check to make sure I'm not missing anything.

Awesome as always guys. :)

HFM viewing 100 frames:
Min. Time / Frame : 00:10:34 - 179,299.8 PPD
Avg. Time / Frame : 00:14:14 - 114,690.4 PPD
Cur. Time / Frame : 00:13:19 - 121,401.3 PPD
R3F. Time / Frame : 00:14:57 - 108,137.8 PPD
All Time / Frame : 00:14:14 - 113,582.7 PPD
Eff. Time / Frame : 00:14:30 - 111,493.8 PPD

Edit: It just picked up another P2686, so I'll leave HFM monitoring it to see how things go. :shrug:

Edit2: HOLY H3LL topping 200k PPD?!? I ~seriously~ have to get my rigs up-to-speed. ...amazing....
 
Hey guys, I've been out of the loop with SR-2 overclocks.. but I'm gonna invest some time to try pushing it more

1) A47 BIOS still recommended?

2) I remember seeing a smp benchmark stress utility, anybody got a link for this?

3) Anybody was able to push their overclock/timings more with NUMA re-enabled(linux)? Sounds possible?


Here's my present overclock:

CPU: 2 x Xeon L5640

198BLCK 1.3125vcore CPU0, 1.3vcore CPU1
1.4vtt
1.815cpuPLL
1.425ioh

RAM: 12GB G.SKILL Eco 1600 1.35V

1066 (1584mhz) 9-11-9-28-1T B2B:AUTO
1.65v mchstrap:auto


Other settings:
numa disabled
PCIE 101
IOH QPI 0 Signal -85
IOH QPI 1 Signal -16
c state, acpi t state disabled
Rest is default..

P6900 : 0:12:49 tpf = 134,222ppd (windows)
Click to expand...
 
404 - File not found said:
Hey guys, I've been out of the loop with SR-2 overclocks.. but I'm gonna invest some time to try pushing it more

1) A47 BIOS still recommended?
Click to expand...

Depends on your CPU - it does not have the ability to unlock uncore - on my x5650/x5660/x5680 you can do much better at 18x uncore for less vtt volts. Apparently on other CPUs this is not the case and A47 is well regarded. But for my CPUs, it is A50 for the win.

2) I remember seeing a smp benchmark stress utility, anybody got a link for this?
Click to expand...

It is on the FAH site somewhere, but it is useless. I ran it for 12 hours once on some settings that then went on to have STOP#124 errors on 6701s. The best test I have found so far is running 2 instances of Intelburntest, very high, 12 threads each, and assigning affinity to each CPU - one to cores 0-11, one to 12-23 etc. See labelling pic below.

Once you can pass 30-60 mins of that, move on to running 12 to 24 hours of 6701s. They are the best unit for showing instability. (you can get a captured 6701 out of Muskys benchmark, and run from there, just kill the benchmark process and the 6701 will run on past 9 frames)

3) Anybody was able to push their overclock/timings more with NUMA re-enabled(linux)? Sounds possible?
Click to expand...

Haven't tried, doubt there is much to be had on my machines, as I never really had a problem NUMA on vs off.

Worth posting this again I guess:
original.jpg
 
All of mine (all with dual l5640s) run the A49 bios. It probably doesn't matter much, though. As MIBW said, the higher end chips benifit from the Uncore adjustment. The L5640s are locked at 18x, so it doesn't help them.

I suspect we will start hearing wierd results with this pseudo-NUMA wrapper thing. Having NUMA on let you hide memory issues, and turning it off would cause problems if you had a bad memory stick. Now, bad memory is going to show up as slower frame times with NUMA enabled. I guess I don't have a recomendation on how to test at this point. I think that NUMA off for the memory and CPU stress tests would be best, but that is a guess.

Higher bclk with L5640s - it depends entirely on the particular chips. More specifically, if you have to crank the Vtt up to 1.375/1.4 to hit 200 bclk, you aren't going to get much over that (and frankly, it isn;t worth trying IMO.) If you pass 200 at 1.35 bclk or less, you have a shot at more. I have one machine that will do somewhere around 210 bclk. My other three are lucky to hit 201 bclk. One bit of warning - the ethernet controllers on SR-2 do not like high bclk. My one machine that will go higher is at 201 bclk now because it drops ethernet connecting starting at around 203. This is also temp-dependant, so with better ventilation, you may get lucky.
 
While I didn't have much luck with my shortly-lived test of tear's Kraken wrapper, I do have chips (x5670s) that, I think, can benefit from the Uncore adjustment. I'm putting SR-2 #3 of 4 together right now, and will try the A50 for the Uncore adjustment + will re-try the Kraken wrapper. More than anything, I'm trying to reduce the average temp + total power draw of the rig.

[H]opefully I'll have more data available soon from my tests and such.

-JT
 
Jmtyra said:
... I do have chips (x5670s) that, I think, can benefit from the Uncore adjustment. I'm putting SR-2 #3 of 4 together right now...
Click to expand...

That is what we like to see....:)

Can you expand on what you mean by having trouble with tear's wrapper? Are you having trouble installing it, or are you not getting the results you expect?
 
Hey :)

More details here on tears page, but I haven't finished the OC on SR-2 #3 to provide him any logs as of yet.

The install was successful; I used your AWESOME install guide + ran through tears instructions after enabling NUMA. The TPF was reporting ~5hrs via HFM, even after I left it going for an hour-ish.

It was probably user error on my part :p but I'll certainly know here in a few hours. Using A50 + Ucore 18x to see what kind of OC I can get and still stay under ~80. When I get close to 80 it makes me nervous. :(

Edit: Apparently you can't run 3x SR-2 boards w/dual OC'd hex procs + your desktop + server + etc off a single 15 amp plug. Kept throwing the breaker. Whoops! I have to wait until another 15 amp SA1810 arrives before I can safely leverage another power circuit and separate some of these power hungry SR-2 boards. This will probably set me back about a week, unfortunately. /sigh
 
Last edited:
Hi guys i am having some issues getting a stable overclock with the 5620's and 48GB of ram. Is there any guide out there that i might have missed in this thread for this CPU? I am playing a bit with my timings but cant seem to get a stable Prime95 run. Any advice or links would be greatly appreciated.

thanks in advance.

Ram i am using is

http://www.newegg.com/Product/Produ...m_re=corsair_vengeance-_-20-233-146-_-Product

Cpu

http://www.newegg.com/Product/Produ...117234&cm_re=xeon_5620-_-19-117-234-_-Product

Mobo - SR-2
 
Speaking of updates, I should receive the last item (SurgeX SA1810) necessary for powering on the 3rd and 4th SR-2 so that I can re-test tears wrapper. Sorry for the delay, g/f's b-day and power circuit related issues.
(Edit: Each SR-2 is eating ~580watts/~5amps, so I ran into circuit breaker problems.)

Hope to have more info soon...
 
Jmtyra said:
(Edit: Each SR-2 is eating ~580watts/~5amps, so I ran into circuit breaker problems.)
Click to expand...

:eek: what are the stats on those, thats 130+ watts more than mines pulling! Even then, I did blow a circuit at home. The same circuit apparently powers about 30 freakin' outlets all over the house so its fairly highly loaded. Switched it over to a different outlet and no problems :)
 
Activate: AMD said:
:eek: what are the stats on those, thats 130+ watts more than mines pulling! Even then, I did blow a circuit at home. The same circuit apparently powers about 30 freakin' outlets all over the house so its fairly highly loaded. Switched it over to a different outlet and no problems :)
Click to expand...
Currently re-doing all of the boards based on MIBW's templates. Running IBT/Linpack pulls ~5.4amp @ 119.3 vac = ~644w. 191x22 = 4.2ghz, 18x ucore ,8-9-8-24 ram, 1.38125v vcore on both, 1.375vtt, 1.35ioh., qpi -90/-16.
Edit: One of the other SR-2 rigs w/these settings is 27% into a P2685 WU @ 10:24 TPF ~183k PPD. This is with tear's wrapper. Was hoping to hit the 200k mark, /sigh.
 
Jmtyra said:
Currently re-doing all of the boards based on MIBW's templates. Running IBT/Linpack pulls ~5.4amp @ 119.3 vac = ~644w. 191x22 = 4.2ghz, 18x ucore ,8-9-8-24 ram, 1.38125v vcore on both, 1.375vtt, 1.35ioh., qpi -90/-16.
Edit: One of the other SR-2 rigs w/these settings is 27% into a P2685 WU @ 10:24 TPF ~183k PPD. This is with tear's wrapper. Was hoping to hit the 200k mark, /sigh.
Click to expand...

Hey, don't worry, be patient - if you get 183k on a 2685 you should clear 200k on 6900/6901/2686/2692s... :cool:

4.2 on 1.38125v is good.

Double check you have C-States enabled in BIOS.

Folding should be closer to 600w - IBT is a wattage beast.
 
may i be worthy said:
Hey, don't worry, be patient - if you get 183k on a 2685 you should clear 200k on 6900/6901/2686/2692s... :cool:

4.2 on 1.38125v is good.

Double check you have C-States enabled in BIOS.

Folding should be closer to 600w - IBT is a wattage beast.
Click to expand...
Wh00t, that sounds awesome. Too bad my OC isn't stable. (It's been coughing on WUs the past ~12hrs.)
I always leave my C-states disabled, so I'll go and correct that now.

Question for you wonderful folks: For an x5670, would you suggest skipping the turbo and aim for a higher bclk = higher mem speed? Also, when my OC fails 'early' it's usually the linpack process crashing (instead of a BSOD). Given the CPU volt range, should I be bumping the vcore a bit?

Thanks as always, guys! :)
-JT

Edit: I'm looking at your template pics here. The Intel(R) C-STATE Tech is shown as disabled. Is that the route I should go, or should I be looking somewhere else? Thanks MIBW!! :D
 
If you are running Linux, it should be enabled. If not, it doesn't really matter.
 
Cool -- I'm totally running Linux (via musky's guide) with tears wrapper. When I enable it, it gives me multiple Cx options...just enable them all?
 
Yes sorry, my templates predate the Linux conversion,

So the changes for Linux are:
  • NUMA enabled - with Tears Krakken Wrapper
  • C-States enabled. (C3 and C6)

I think vcore and possibly IOH are too low. If you run 2 instances of linpack, and assign them to each CPU, then you can see which one needs more.

Yes you will get a bit more performance running at 200 x 21 (turbo off), but you will face greater stability hassles. One of my SR2 boards can do 205max, another 201, so you would be close to the edge. My gut feeling is that the case for doing so with Linux/affinity wrapper is diminished, as each CPU sticks to it's own memory now, whereas before the inefficient juggling put a premium on memory performance.

Linux wrapper results at 4.3Ghz on 6901 benchmark:
SR2B - x5660 - 187 x 23 (21+2 turbo) - 9:35
SR2C - x5680 - 205 x 21 (21+no turbo) - 9:30

There used to be a bigger difference on Windows, now it is down to 5 sec = probably not worth the extra hassle.
 
To answer your other question, CPU speed is almost always better than RAM speed. As a general rule shoot for the config that will net you the fastest CPU speed.
 
Question: Does having two Noctua DH-14s on an SR2 mounted vertically in a case bend the board over time?
Answer: Why yes it does. (Son. Of. A. Bitch)

Last night I noticed SR2#1, - 8 months old, had dropped a stick of ram: 22GB.
I rebooted at optimised defaults - still gone.
Gently reach inside case and press on ram, boot again, 16GB!
repeat process - boot again - 12GB!
Remove case from rack (oiy the shenanigans that took:rolleyes:), place on side, pop all sticks of ram, and reseat, boot: 24GB! Phew.:D

Now I have to rearrange my rack to accomodate a truly massive case horizontally - a sleepy SR2 that just needs a litttle lie down. This board is too big, and the DH-14s are too heavy for mounting upright over a long period without counterweights. Naked flat on the floor is so much better.

But all things considered I think I got off lightly.
 
Sounds scary. Running with the heat sinks standing vertically makes the most sense for any system, less force on the PCB. One more reason I should get a rack instead of using tower cases.
 
Wow thanks for the warning. Makes me nervous about my hyper 212 and TRUE coolers on ny Sr2
Posted via Mobile Device
 
I don't know that I'd worry too much about Cogage or Hyper 212+ coolers. They're not that heavy compared to the monster Noctua DH-14s. Of course, that assumes your mobo is solidly mounted to a tray. I haven't noticed any issues with the Cogages in my Lian Li, but I'll double-check when I get around to mounting the fans in the top panel.

That said, I'm very happy with the MM U2-UFO Horizon case layout. Cooling is still a bit of a challenge (even with 8 fans) but the TRUEs are very stable.
 
SazanEyes said:
Of course, that assumes your mobo is solidly mounted to a tray.
Click to expand...
Uh, mine isn't exactly solidly mounted. You can see my situation in this image provided by Zero2Dash. I ran a zip tie through the top right hole that keeps the mobo in place, but I never did anything about the screw in the middle of the top edge.

And I have the real TRUE - Thermalright Ultra Extreme which weighs 790g before fans (I have two) as opposed to the cogage at 670g with one fan or the Hyper 212 at a svelte 626g with one fan (I added one to this as well). I figure fans are about 125g each, so my TRUE weighs about 1050g and my Cogage weighs about 750g. Ugh.
 
I think also having all DIMMs slots populated is going to show problems more often... I am just hoping there is not more serious damage.

@Jebo - I think this means Tobit is getting back in the SR2 game maybe? The Purple Orchid color stuff is just to weird us out. ;)
 
may i be worthy said:
I think also having all DIMMs slots populated is going to show problems more often... I am just hoping there is not more serious damage.

@Jebo - I think this means Tobit is getting back in the SR2 game maybe? The Purple Orchid color stuff is just to weird us out. ;)
Click to expand...

He is going G34 I'd bet.
 
jebo_4jc said:
And I have the real TRUE - Thermalright Ultra Extreme which weighs 790g before fans (I have two) as opposed to the cogage at 670g with one fan or the Hyper 212 at a svelte 626g with one fan (I added one to this as well).
Click to expand...

Yeah, I have all three coolers. This is one reason my TRUEs are in the horizontal SR-2 and the Cogage in the tower SR-2. I had problems with the motherboard flexing back when I first got it and was running it on top of the box. It's just so big, it wants to bend.
 
Anyone seen this weird delay time shutting down the core (delay 1) and getting new WU (delay 2). My sr2 is using Ubuntu with and w/o the Affinity Wrapper (same delays). When I tried it on FC 14, the delay is cut in half but still add up to 30 minutes worth of waste. During delay 1, the hard drive activity LED is on - it seems like it's accessing the disk. I have tried both the USB Flash drive and internal Sata II (several different drives), same delay time. Tried partition type: FAT32, Linux Ext4, XFS - non seems to solve this delay problem.

I tested copying the WU result file (about 100MB) on the SATA II drive and it took less than 1 second. Could it be the write delay cache setting? Doesn't make sense to me. Any idea?

Code: 
[13:49:55] Completed 250000 out of 250000 steps (100%)
[13:50:05] DynamicWrapper: Finished Work Unit: sleep=10000
[13:50:15]
[13:50:15] Finished Work Unit:
[13:50:15] - Reading up to 56480736 from "work/wudata_03.trr": Read 56480736
[13:50:15] trr file hash check passed.
[13:50:15] - Reading up to 45766324 from "work/wudata_03.xtc": Read 45766324
[13:50:16] xtc file hash check passed.
[13:50:16] edr file hash check passed.
[13:50:16] logfile size: 192199
[13:50:16] Leaving Run
[13:50:20] - Writing 102609207 bytes of core data to disk...
[13:50:21] ... Done.
[[COLOR="Red"]14:29:20[/COLOR]] - Shutting down core
[14:29:20]
[14:29:20] Folding@home Core Shutdown: FINISHED_UNIT
[14:33:34] CoreStatus = 64 (100)
[14:33:34] Sending work to server
[14:33:34] Project: 2685 (Run 4, Clone 19, Gen 52)


[14:33:34] + Attempting to send results [March 31 14:33:34 UTC]
[14:41:56] + Results successfully sent
[14:41:56] Thank you for your contribution to Folding@Home.
[14:41:56] + Number of Units Completed: 3

[[COLOR="Red"]14:58:16[/COLOR]] - Preparing to get new work unit...
[14:58:16] Cleaning up work directory
[14:58:16] + Attempting to get work packet
 
vcdz382 said:
Tried partition type: FAT32, Linux Ext4, XFS - non seems to solve this delay problem.
Click to expand...
This is a known problem with both EXT4 and XFS. The preferred file system on Linux is EXT3.
 
Tobit said:
This is a known problem with both EXT4 and XFS. The preferred file system on Linux is EXT3.
Click to expand...
Thanks for the info. I'm installing Ubuntu on Ext3 now. BTW, where is the UserId stored in Linux? I stopped the WU midway and switched OS and looking to resume after the new OS is up. Is copying the entire "fah folder" good enough? (In windows, I would need to import the registry key to set the UserId).
 
You must log in or register to reply here.
Back
Top