How to Tune Quad E7-4850 System

[plext0r]

I'm testing F@H on a Cisco UCS B440 M2 system. This system has 4 E7-4850 CPUs and 256GB 1067MHz RAM (4 16GB DIMMs per CPU). I've made sure NUMA was enabled, node interleaving disabled, Turbo & Speedstep enabled, all C-states disabled. I'm running Scientific Linux 6 native (no VMware) with their latest kernel (2.6.32-based).

I installed "thekraken" to handle CPU affinity as well. It appears to be working correctly.

With 80 2GHz cores folding project 8101, I'm seeing TPF of ~15:00. I would think I should be closer to 10:00 per other forum posts, but since I do not believe I can overclock, I'm not sure what tuning can be done. Was I wrong to disable all C-states?

I've tried some of the CPU tunings (Enterprise, HPC, High-Throughput) which disable or enable prefetch and data reuse. It didn't seem to affect TPF very much. I'm currently running with prefetch and data reuse disabled (Enterprise).

Since the BIOS provides no provision for overclocking, is there anything I can do to decrease the TPF? Thanks for any pointers.

22:55:22:WU00:FS00:0xa5:Completed 20000 out of 250000 steps (8%)
23:10:56:WU00:FS00:0xa5:Completed 22500 out of 250000 steps (9%)
23:26:30:WU00:FS00:0xa5:Completed 25000 out of 250000 steps (10%)
23:42:03:WU00:FS00:0xa5:Completed 27500 out of 250000 steps (11%)

 

[extide]

Make sure that disabling all those c-states doesn't also disable turbo, you should get a full bin of turbo at all-core-turbo.

 

[jebo_4jc]

I would personally leave CPU power settings at default in an effort to ensure turbo is engaging properly.

 

[extide]

I would personally leave CPU power settings at default in an effort to ensure turbo is engaging properly.

I would do the same.

 

[Patriot]

need to set a performance mode in linux...
its been awhile since I tweaked mine...but turbo probably isn't running on your monster.

(I had an E7-4870 folding for a bit)
ill do some digging and brb

-ACPI needs to be on... and the acpi module needs to be loaded in the OS.
and scaling governor set to performance...

So...

edit: /etc/sysconfig/cpuspeed

change

GOVERNOR=ondemand

to

GOVERNOR=performance

and

modprobe acpi

taken from here....its for rhel5 but I think is should work for a rhel6 based distro.
http://murty4all.blogspot.com/2010/11/changing-cpu-frequency-scaling-governor.html

 

[quickz]

Where did you see 4p E7-4850's TPF of 8101 should be closer to 10:00?
I think this is impossible. The best performance for E7 I have ever seen is 11:27 on a 4p E7-8860. If we scale it to 4p E7-4850 it's only around 13:00. See:

http://hardforum.com/showthread.php?t=1716471&page=2

However, your rig is still 15% slower than it, I suggest you download a turbostat or i7z to see if turbo boost is working properly for your rig.

I'm testing F@H on a Cisco UCS B440 M2 system. This system has 4 E7-4850 CPUs and 256GB 1067MHz RAM (4 16GB DIMMs per CPU). I've made sure NUMA was enabled, node interleaving disabled, Turbo & Speedstep enabled, all C-states disabled. I'm running Scientific Linux 6 native (no VMware) with their latest kernel (2.6.32-based).

I installed "thekraken" to handle CPU affinity as well. It appears to be working correctly.

With 80 2GHz cores folding project 8101, I'm seeing TPF of ~15:00. I would think I should be closer to 10:00 per other forum posts, but since I do not believe I can overclock, I'm not sure what tuning can be done. Was I wrong to disable all C-states?

I've tried some of the CPU tunings (Enterprise, HPC, High-Throughput) which disable or enable prefetch and data reuse. It didn't seem to affect TPF very much. I'm currently running with prefetch and data reuse disabled (Enterprise).

Since the BIOS provides no provision for overclocking, is there anything I can do to decrease the TPF? Thanks for any pointers.

22:55:22:WU00:FS00:0xa5:Completed 20000 out of 250000 steps (8%)
23:10:56:WU00:FS00:0xa5:Completed 22500 out of 250000 steps (9%)
23:26:30:WU00:FS00:0xa5:Completed 25000 out of 250000 steps (10%)
23:42:03:WU00:FS00:0xa5:Completed 27500 out of 250000 steps (11%)

 

[plext0r]

I would personally leave CPU power settings at default in an effort to ensure turbo is engaging properly.

Done. I re-enabled all C-states.

 

[plext0r]

need to set a performance mode in linux...
its been awhile since I tweaked mine...but turbo probably isn't running on your monster.

(I had an E7-4870 folding for a bit)
ill do some digging and brb

-ACPI needs to be on... and the acpi module needs to be loaded in the OS.
and scaling governor set to performance...

So...

edit: /etc/sysconfig/cpuspeed

change

to

and


taken from here....its for rhel5 but I think is should work for a rhel6 based distro.
http://murty4all.blogspot.com/2010/11/changing-cpu-frequency-scaling-governor.html

I tried installing cpuspeed and using the performance governor. My CPU MHz went from 2000.089 down to 1996. As soon as I unloaded acpi_cpufreq kernel module, my MHz went back to 2000.089. I think you guys are right about Turbo not engaging so I'll investigate further.

 

[plext0r]

Where did you see 4p E7-4850's TPF of 8101 should be closer to 10:00?
I think this is impossible. The best performance for E7 I have ever seen is 11:27 on a 4p E7-8860. If we scale it to 4p E7-4850 it's only around 13:00. See:

http://hardforum.com/showthread.php?t=1716471&page=2

However, your rig is still 15% slower than it, I suggest you download a turbostat or i7z to see if turbo boost is working properly for your rig.

I may have been mistaken with "closer to 10:00" but I thought 15:00 was slow. Thanks for the pointer. I don't think Turbo is engaged or I'd see 2.133 MHz in /proc/cpuinfo. I'll look into those two utilities and see if I can use of them easily. Thanks for all the tips guys.

 

[plext0r]

I compiled the latest turbostat and it appears to say my cores are running at 2.13GHz. Here's a snippet:

pk cor CPU    %c0  GHz  TSC    %c1    %c3    %c6   %pc3   %pc6
           100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   0   0 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   0  40 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   1   1 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   1  41 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   2   2 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   2  42 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   8   3 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00
 0   8  43 100.00 2.13 2.00   0.00   0.00   0.00   0.00   0.00

I disabled all hardware prefetch and fetch extra cache line last night, but it has not affected TPF.

 

[Patriot]

15 may be right...

2000.089 means over 2... though I thought mine read 2400.1 when turbo was on...
it will never show the actual turbo without using something like turbostat.

Thats interesting... I wonder if they changed the modules around from rhel5 to 6 or from rhel to scientific...

I will see if I can find my 8101 tpf for that box... it was no 4p 2011 ...and my 4p g34box creamed it...

mmm looks like @ 2.4 you would be at ~11:30 tpf on 8101

you can double check to make sure top shows your cpu usage at 8000%
and that you don't have any windows running... (this includes top in terminal close it after checking)


....random thought... make sure your system isn't being power capped or thermally throttled... bios power state to max performance and fans to that as well...
These types of rigs tend to draw quite a lot at peak load.

And.... tpf on a paused unit cannot be trusted... so do some changes and check on it tomorrow.