Haswell-E 5960X 8-core system, OC assistance appreciated

Hello everyone. I've recently built a system based on the X99 platform and am looking forward to overclocking it. Unfortunately, I've run into a lot of conflicting instructions/suggestions for how to go about doing so. I'd appreciate some guidance. Here are my system specs....

Intel 5960X Haswell-E 8-core CPU
Swiftech H320 X2 Prestige AIO CPU cooler
Asus ROG Rampage V Edition 10 X99 motherboard (latest 1701 BIOS from June 2017)
Corsair Vengeance LPX 16gb (4x 4gb) DDR4 2666mhz RAM
Samsung 960 EVO M.2 NVMe SSD
Nvidia GTX 1070 8gb - Asus ROG STRIX OC model GPU (For the moment, perhaps replaced with AMD Vega when it arrives)
EVGA SuperNova G3 1000w PSU
CaseLabs Mercury S8 case (Two tone, black exterior white interior) w/ fans from Noctua and Corsair (mag-lev) + DEMCiflex filters on intakes
Extra hardware (At least 1 SATA SSD, 1 SATA HDD, possibly other items).

My CPU OCing experience on the Intel HEDP in depth is from the old Nahalem/Westmere i7 generation, so I am out of date on the exact settings and whanot that need to be changed here. Lots of advice regarding overclocking a Socket 2011-3 CPU seems to vary significantly based on if it came from the earlier part of the X99 release schedule versus the later/current one, Haswell-E vs Broadwell-E ,and how many cores you have. Given the specs above I think I have a pretty solid system to go for a mid-range OC at least. I am using my Asus Rampage V Edition 10 with the latest 1701 BIOS from June 2017, so any specific titles/settings I quote are with this in mind. As one would expect, this board gives a ton of OC options which is an excellent feature. The exact mechanics by which one should achieve this kind of OC, in my case, I'd appreciate some assistance.

My final goal is the highest average OC my hardware will accept; I want to get as much out of the chip as possible before hitting that point where there are greater risks , diminishing rewards, and lots of tweaking of minutia just to get 100mhz more etc.. From what I've read, a Haswell-E 8-core chip should be OCable from 4.4 to 4.7ghz on good air or AIO liquid easily, based on the "silicon lottery". Ideally, I want to propagate my OC across all 8 cores (and 16 threads). I am unsure if I can leave Intel SpeedStep and similar technologies active - it seems beneficial to lower temps and whatnot if one is able to do so while still having the possibility of getting to the full OC when neeeded. Likewise, I am unsure how the Turbo feature acts in respect to Haswell-E and OCing. My understanding is at stock the chip will clock up 1 or 2 cores to "Turbo" speeds, so I am guessing that if you are OCing that this functionality is disabled because we are already surpassing the stock Turbo specs? However, on my RVE10's bios when on the "Extreme Tweaker" page it shows various "Targets" which will result from the settings you have input, and one of them is "Target CPU Turbo-Mode Frequency" which seems to be the OC'ed frequency. Does this mean that in this case, Turbo will apply to max CPU frequency across all cores instead of only a couple of them?

Before we get started, I'd like to ask which tools and tests should I use to confirm stability. Things have come a long way from Prime95 (which I hear is now capable of basically damaging your processor due to power drain), Linpack and the like . I was thinking of using AIDA64 for both monitoring and stress testing (All components), Asus ROG RealBench for stress testing (all components), CoreTemp for temp monitoring, HWMonitor for temp monitoring, and/or MemTest64 (TechPowerUp's program) if I need an additional memory test besides those in AIDA64 + RealBench? Is there anything else I should be using and/or something I should NOT use?

That said, now comes the matter of actually picking the settings. Except for the things I mention, assume most everything else is left on default/auto.

"Ai Overclock Tuner" - Taking this off the default Auto setting grants either "Manual" or "XMP". XMP seems like a good place to start as it seems to automatically populate some settings, especially the RAM settings and also opens up the BCLK/CPU Strap.

XMP - This seems to have 2 presets for RAM, one for the 2667mhz at 1.20 (if I recall) and another at 2800 at higher voltage of 1.35. I figure I ought to start on the 2667mhz because the kit I have is actually rated for that. This might take care of quite a few settings if the profile is worthwhile. If not, perhaps I should manually tweak later?

CPU Strap - This seems to be a very important setting that affects everything else out there pretty much. Earlier I saw some articles suggesting a 125MHz frequency, but more recently produced ones suggested that if your system will take it stick with 100MHz? XMP seems to set it at 100MHz by default so I figure I'll leave it there unless necessary. I should mention that the XMP profile sets this at 100mhz by default and also sets the BCLK at 100.0; I'm to understand you're supposed to keep the CPU Strap and BCLK close to each other as possible.

CPU Core Ratio - Sync All Cores. There is also an Auto option as well as one to populate it by core use etc. Putting it to Sync All Cores I am gathering keeps all cores running at the same max speed. This seems to be the setting that may be the most useful to ensure that Turbo not function as the Intel default with one or two cores having a higher potential maximum, but instead setting them to all the same speed.

1-Core Ratio Limit - With Sync All Cores above, setting this first core propagates the same ratio to all 8. It starts at 30 if I recall by default. If I want to start at 4.4ghz to start, then I'll set it to 44 . A good choice?

Min CPU Cache Ratio - Apparently this has to do with overclocking the cache. By default, it is set at 31. What should I change this to, if anything?
Max CPU Cache Ratio - Just like the above, this is by default at 31. I am unsure if this is something that it is worth changing or not. I've read that changing these ratios in order to be in parity with my CPU Core Ratio, is ideal - but may require tweaking of other voltages?

CPU Core Voltage - If what I've read is correct, if I want to accomplish a 4.4ghz+ OC, I'll need to add the voltage to 1.300v (but unlikely any higher, even on AIO water unless absolutely necessary)? I've also read some chips will handle it on 1.250. By default, the setting is on Auto and it reads off the current voltage at 0.950V - which I assume is the low end of the scale thanks to just sitting at the BIOS, with little stress on the chip. If i was to set a say, 1.300V limit, would this be indicative of a maximum voltage that the system would dynamically use up to when necessary (which seems desirable to keep heat down etc), or would it push that amount through all the time? What kinds of setting and what sort of options should I choose here?

CPU Cache Voltage and/or Any other voltages - Not sure if any others need to be tweaked. ?

Turbo Boost - As I reference above, I am unsure if in this case it is best to leave Turbo Boost on, or turn it off, and exactly what ramifications this will have.. The difference between a system with a 4400mhz across all cores without Turbo, and one with Turbo enabled, I'd be curious to know. Many OC videos I've seen have left Turbo on and didn't mention it at all.


There are also lots of other settings and menus out there (ie the entire list of DRAM timings and whanot, but I assume that XMP will set these accordingly and save me some time. I would be willing to tweak a little after the fact to tighten them), but from what I've read that unless you're truly going for an extreme OC or are trying to figure out a problem, most of these can be left stock. Are there any I am missing and are either required and/or would be potentially useful?

Thanks for the advice thus far - I am hoping to get a good, solid OC out of this Haswell-E chip and have it, along with the board and RAM pairing, make for a thriving OC'ed base to my new rig.

Quick and dirty - set the multiplier to 42 for a 4.2GHz clock speed and the VCore to 1.3V. Run a CPU stress test of your choice, and if stable up the multiplier 1 at a time. My 1660V3 (5960X equivalent) only got to 4.3GHz across all cores at 1.35V - a bit of a overclocking dud, yet yours might be a good one reaching 4.5 or even 4.7GHz. From what I've read you do not want to run more than 1.35V or risk frying your chip really early (even if your temps are good).

RAM speed isn't going to increase performance that much, but you will likely want to keep the BCLK at 100. Some frequencies will up that to 125 which causes additional strain on the CPU's IMC and can cause instability. For my EVGA board (it may be different for yours) 2133, 2400, 2666, and 3200 will all run at 100 BCLK. Really, you wont see huge gains going from 2666 to 3200 Unless you have very specific work loads that take advantage of it, and I'm not talking about gaming.

Ah, I suppose I could go a bit lower to 4.0 or 4.2 just to be safe, to see what works at 1.3V. My goal would be at least 4.4, and better 4.5 as I read that is an 'average" 5960X chip according to both studies Asus did and aftermarket enthusiast overclockers, but its always good to be safe. I certainly won't go above 1.35. When it comes to these voltages, are these basically maximum voltages being set for when the core needs it, yet the system will automatically lower it when it isn't necessary (possibly contingent on SpeedStep, C-States etc.?) Or are they basically forcing it to run at that voltage all the time? Just to be clear, I intended to put that multiplier equally across all cores.

Any input on cache? I've read that once you get CPU stable if possible its worth trying to set the Cache at the same multiplier as your CPU cores (perhaps both Min and Max cache multiplier both set at the same, 40-47 etc) , but I am unsure what voltage changes (if any) are required or if it can basically be left at auto.

The one big question I had was about Turbo Boost - it seems that most discussions of OCing on X99 don't talk about Turbo Boost (or SpeedStep for that matter) at all, seemingly leaving it enabled (ie I saw a Linus Tech Tips Haswell-E OC guide where he leaves it on and doesn't mention it at all). On my Asus board, it shows projected CPU speed, RAM speed etc.. in the BIOS based on the tweaks you're making and if you leave Turbo on it shows your CPU speed (based on multiplier) as something like "Max Projected CPU Turbo Speed" versus "Max Projected CPU Speed" if its turned off. I don't know if, for this context when we're OCing above what the stock Turbo speed would be and across all cores instead of only a few, it makes any difference at all?

I do hope to keep the BCLK and CPU Strap at 100, and read that as you say, at 2666 it will be possible to do so. On my board there's an option to enable XMP for RAM (and a few other things it affects) as a profile, so I wonder if its reasonable to just use that, though I could always set things manually too.

Thanks!

If you disable speed step, you won't get turbo boost states - which is what you are actually overclocking when you set a higher multiplier than the stock clocks.

Voltage depends on what you set - I use the offset method, 1.3V and with a +50 offset, which results in ~1.35V pulled when the CPU needs it. When the CPU isn't running at a high speed it will pull less voltage, which is about .990V right now at 2.7GHz. Unless you set your Windows power profile to keep the processor at 100% full frequency state all the time, which can be helpful with really high overclocks.

XMP profiles do have issues on some RAM / Motherboards - I suggest dialing in your CPU OC for stability first, then testing whether or not the XMP profile makes things unstable (or vice versa). You can always manually set the timings, voltage, and speed in your BIOS should the XMP profile give you issues. Stock RAM speed for the 5XXX series is 2133MHz.

For the CPU multiplier, all core is the standard. With mine, I can set all core to 43 and 6 and below to 44 and still be stable. However, I never see any CPU loads that trigger 4.4GHz - which is likely why setting all core multiplier (turbo) is the de facto standard.

If I'm rambling, and also not answering everything that you've asked, I apologize - I'm about 3 mixed drinks in and am feeling good

On my board, there are both options to enable/disable SpeedStep and Turbo Boost independently. I was hoping to keep SpeedStep and from what I read that won't be a problem. However, I guess what I'm trying to figure out is if I put a 44 multiplier on every core, what will be different with Turbo boost on versus Turbo boost off (yet SpeedStep on for both) when in both cases, it suggests a 4.4ghz clock rate. GIven that speedstep is on in both cases, logically it suggests that the CPU will automatically downclock/downvolt when the system isn't under stress. Given that we've already surpassed "stock turbo" behavior of it clocking up beyond the normal 3.0 limit to 3.5 on certain cores under workload if the CPU detects it is cool enough, what exactly will Turbo on vs Turbo off mean in respect to that 4.4ghz clock rate ? Will it mean that with Turbo on will only clock up to 4.4ghz under certain workloads, yet Turbo off it will always be at 4.4ghz regardless of workload? Or something different?

Ahh all right. I'll have to look at the varying methods of voltage specification. Besides "Auto", I seem to remember there was both setting a voltage directly (ie 1.3V) which I suppose acts as the max voltage, and also an offset option that I didn't see discussed very much in tutorials. So to confirm, using offset means setting a baseline voltage, plus an amount it can scale up to if necessary? How is this different from say...setting at 1.35 directly and knowing that if the processor doesn't require it (ie not working as hard etc) it simply won't scale up that much? Or is there different behavior?

Thanks for the idea about XMP. I'll just tweak the CPU itself first and then when I get to the RAM, try XMP to see if it works. If not, I'll just go manual.

What order should I work on my the OC items? Clearly the CPU itself should come first and the eventual RAM tweaking/settings come later, but what about something like CPU Cache? Is it worth messing with it at all? I remember reading that its a good thing to see if you can get the CPU Cache at the same rate (by multiplier, with Min and Max both set at 44 etc) as the CPU if possible, but I don't recall much discussion if this will require an increase to the cache voltage specifically and if so, what numbers should be used as "safe" and what is the likely "don't cross this point"?

Heh, thanks for your help thus far, 3 drinks in or not! I welcome input from you or anyone else willing to give a hand/experience

You've got speedstep right, yet turbo boost is literally what you are overclocking. If you disable turbo boost, it should limit your CPU to 3.0GHz. At least that's how it works on my board. Speed step allows the processor to downclock when there isn't enough of a load on it, turning it off will make the CPU run at max clock all the time.

Auto voltage tends to add too much. Fixed voltage is exactly that - 1.35V whenever the CPU goes max multiplier - yet the voltage can still be automatically lowered with speedstep enabled. Offset just let's the CPU pull more voltage as needed to maintain stability if it needs more than the baseline set (1.3 in my case, for ~1.35 max). Not all workloads are the same, so this allows the CPU to take what it needs up to about where you specified - it allows for a somewhat reduced power draw, and theoretically a longer lifespan for the chip. Offset usually isn't the best choice with really high overclocks - 4.4GHz isn't that high, so you'll be fine with either.

As you've already stated, a CPU overclock will net the most gains with almost everything that you do. Yet overclocking the CPU cache doesn't net much except in very specific workloads and benchmarks (it's on the chip information passing, which is already insanely fast) - I leave it at auto on my desktop rig since setting even a 40X will cause a no boot with a 43X CPU multiplier, and on my 6820HK laptop I set it to 38 from a stock 36 when it's CPU milultiplier is set to 42 (much higher requires a lot of extra voltage which causes a lot of extra heat for almost no difference and a much higher rate of instability). RAM speed also has similarly small gains - you get more out of tighter timings for most tasks from all that I've read, yet dialing in stable tight timings can be tricky.

That said, if your CPU cache can run at 40 or even 44 without too much extra voltage and heat and you can run your RAM faster or on tighter timings (and stable!), I say go for it.

Conversely, you can get an OC that you're comfortable with and don't feel the need for more, like 4.4GHz, and then you could lower the VCORE a little at a time to see how low it will go before becoming unstable, then add a bit to get back to being stable. Some folks can run 4.4 or even 4.5GHz at 1.2 or 1.25V, which can keep temperatures nice and cool.

Oh now that's new, so disabling Turbo Boost basically eliminates the possibility to overclock - so it isn't specifically related to the model of how stock Intel processors tend to boost one core or another if there's "room" in terms of temperature/volts, but any ability to clock over the hardwired "norm" of 3.0 in my case, even manual overclocks? The only question I'd have here is that, if I turn off Turbo Boost (leaving SpeedStep on) , and set up my CPU ratios to 44 etc.. my BIOS states the"Projected CPU frequency - 4400mhz" or some such, whereas i do the same with Turbo Boost turned on it says something like "Projected Turbo Boost CPU Frequency - 4400mhz". Perhaps it wouldn't "actually" work, but I have to wonder why the BIOS would list that differently if it wouldn't. Perhaps your experience is different because it is on a Xeon which if I recall, does not actually support overclocking (though they do support turbo boost)?

I think I have enough to get started however, thank you! Just a couple of questions on other settings. First of all, do you have any idea if turning on some of the Virtualization settings in BIOS (which seem to be off by default) , like for Virtualization support under the CPU, VT-d etc.. will affect anything one way or another? I may at some point end up using a virtualized guest OS (ie Windows virtualized within Linux, Linux within Windows etc) and if these will give better performance/access to virtualization (especially things like GPU passthrough) for VirtualBox / VMWare etc... then it seems like a good idea. However, they're turned off by default so that makes me think I may be missing a reason for it, despite not seeing these settings mentioned regarding OCing or stability in general.

Lastly when it comes to stress testing, do you think using AIDA64, Asus RealBench, and possibly Intel's Xtreme Tweaking Utility (pretty sure this has s stress test?) is sufficient for the CPU and system as a whole? Likewise Memtest64 for testing the RAM if it needs its own run. How long do you consider running the average stress test to consider it stable for high performance gaming and whatnot? In the old days people talked about running certain things for 12+ hours or even a day, but I now see more discussion that even an hour or two on some of the better utilities seems reasonable. Likewise, with those with many settings like AIDA64, the stress test has options for CPU, FPU, Cache, RAM, and GPU. The last 3 obviously don't seem to be necessary unless testing those particular elements or wanting to do a final "everything all at once" test, but I've read that the first 2 - CPU and FPU are for testing the CPU performance. The "CPU" option is the standard way and probably the first one that should be run, but I've heard that turning on "CPU+FPU" is basically going to simulate a MUCH higher than typical AVX-enabled load that is going to strain the voltage and temps a lot more. Is this worthwhile/safe for a Haswell-E? I read that these, unlike prime95 and whatnot won't push things "too" far and do damage, but its still hefty. Finally, temps during stress testing - I remember reading somewhere that 85c is a good max-load-temp target for an OC'ed Haswell-E processor but is that 85c per core sensor, or 85c for general CPU sensor which is usually cooler?

Thanks!

85C max per core is an excellent temp to shoot for. The CPU will be much hotter when testing AVX loads, which is why many enthusiasts say to stay away from the >26.6 version of Prime 95, which uses AVX. Now, some folks need to test AVX stability (rendering, video editing), but your average gamer does not. Real Bench and Aida64 are both excellent options for stress testing, and I've read that many people prefer Real Bench these days (which I used - I also tested with version 26.6 Prime95).

Length of testing? I've found that you may or may not be lucky with short stress tests. Hell, even stable for 48 hours straight might not be 100% stable when the CPU load fluctuates (that's why some people set 100% minimum processor state, and / or disable speedstep). My last go was stable at 4.4GHz when stress testing, yet locked up in games every once in a while. I backed off 1 multiplier because I didn't want to put any more voltage on the CPU.

As to the turbo boost on your motherboard - it very well could act differently due to my board or my Xeon. Try it out, you won't damage anything. Also, the V1, V2, and V3 Xeon's are multiplier unlocked - Intel locked the V4s down.

As to virtualization... that's not an area that I'm familiar with, yet I do know that some features are Xeon specific and may not be enabled on your 5960X. Other folks will need to chime in on that.

All right, after a little bit of stress testing I think I've come pretty close to getting things squared away. I've managed to bring my CPU to 4.5 ghz at 1.3v - AIDA64 stress tests, including FPU option, plus I've tried Intel XTU too. . Alas I can't use RealBench yet as I'm using an ancient NV GTX 260 as a placeholder GPU and apparently openCL doesn't support it so Real Bench won't run. I'll have to use it as my final stability test once I drop my 1070 into the new PC. Sadly, when I push it to 4.6 it seems the test fails (but not crashing my whole PC thankfully). I'm wondering if its worth it to push it much further - from what I've read that 4.5 at 1.3v means basically I have a "good" chip, and 4.6 or higher would mean a "great" one or better , silicon lottery style. If I was to try getting 4.6 stable, Temperatures during the test at 4.5 are no problem at all whithout turning on FPU - when I do turn on FPU they stay below the danger zone, but a couple of cores do get close to 85C , rarely and briefly as it might be. Oh by the way, my "CPU Package" sensor is actually the hottest one out there, which during AIDA64 CPU+FPU apparently hits as high as 90C (rarely and often not for long) which is above both the CPU and the per CPU Core sensors at that time - should I be concerned about this? No matter what, nothing has ever throttled according to AIDA64 or failed a test because of temps etc.

A couple of questions about AIDA64 Stress Test. there are options to stress the CPU, FPU, Cache, RAM, and GPU. I started by running just CPU by itself, then CPU + FPU for the "most stressful" CPU test with AVX and whatnot. However, it appears to me that if I enable CPU + FPU + Cache + RAM - it actually doesn't stress the CPU as hard as one would expect? That is to say, it acts more like i just used CPU + Cache + RAM together, because the fans/heat increase of the FPU test won't seem to actually increase if I enable other options besides CPU? Is this by design or is something else going on? I figured I'd run CPU + FPU + Cache + RAM as one of my last AIDA64 stress tests to make sure the whole system was working together, but I gather that perhaps it doesn't enable the FPU test loads?

I should mention that I've tested my 4.5ghz at 1.3v OC while also both enabling Intel Virtualization Technology and Intel VT-d in my BIOS - it doesn't seem to make a difference so far as I can tell! I also turned on my RAM to XMP specs - 2666mhz @ 1.2v and standard timings for this kit (CL16 etc. I may try to tweak them a bit tighter if need be. When I upgrade to 32GB that will be much newer RAM at 3000/3200mhz or better at tighter timings,, but I'lll use this for now). BCLK and CPU Strap are still both at 100.0 and 100mhz.! So the only thing I have left to do is figure out the Cache

At the moment, thanks to XMP my Cache is set to a multiplier of 31 by default, min and max both, and the default 1.2V. It seems to be stable here (AIDA64 stress test Cache mode) which is no surprise. I OCing CPU Cache, should I set both the Min and Max to same multiplier, or do I only basically set the Max to the new multiplier and leave the Min at something lower and/or default? Not sure if Speed Step / Turbo etc.. affect Cache in terms of what governs cache speed at a given time?

Now I've heard 2 conflicting opinions when it comes to Haswell-E Cache OCing. Some say that you should try to bring to to parity with your OC if you can, or close to it. Others say that if you OC over 4.0ghz its practically impossible unless you have a golden chip and/or add lots of voltage/tweaks etc, so its best to bring Cache to 3.8 - 4.0ghz at best without going over, which will be much easier. As you said before I know Cache OC is not a major feature, but I'd be willing to give it a bit of an increase for the hell of it without having to work too hard and/or add too much heat/voltage/instability to the rest of the OC. If I was to take it to 4.0 (lets say), do you think its still best to leave the voltage at auto or do you know if it should be increased to a particular number? Will I need to increase any voltages besides CPU Cache voltage? When testing Cache stability, should I simply use the AIDA64 "Cache" option by itself? Or CPU + Cache?

I can't speak to Aida64, someone else might be able to chime in there.

4.5 at 1.3V is good! And I do agree that your temperatures are right about where I find comfortable. Better than my Xeon equivalent. As long as it is stable, you really should be happy with it. You might be able to lower the voltage a bit and maintain stability - if you already haven't tried that out. It won't help you get a better OC but it will help keep temps down.

As far as min/max for cache - that's a new one to Me, as I haven't seen that option on my X99 Classified. Cache speed will step down with speed step. At least I know it does on my Skylake 6820HK, so I'd wager it does on the 5960X as well (I've never checked with my 1660V3).

I've also read about trying to being the cache speed up to parity, or as close as you can, to the CPU multiplier. In practice you'd need a really good chip to do so because with the two chips that I've tried on, neither can stably attain over a 38. I've not messed with anything other than auto voltage for the cache, so that might be what was holding my chips back. Someone else, or even someone on your motherboard manufacturer's forums, could help you out more than I can on that subject.