Haswell-E, Is your overclock Prime95 28.x AVX2 stable?

[sblantipodi]

As title.
Asus suggest to not use the latest Prime95 small fft test with Haswell-E.

We should not use small fft but what about the default test?
I found that test incredibly reliable to see if the system is stable or not.

I know that a system can be 100% stable and crash with other types of loads but now let's try to concentrate on Prime95 28.x.

Do you use it?
If yes, is your overclock stable to the default blend test? At what frequency/voltage/ram speed?

 

[jrweis]

Used Prime 95's default test on recent Asus X99 Deluxe build with an 5960X processor and 64 GB of G.Skill 2400 memory. Test was run will memory set at XMP profile, and all other settings stock as this was for a high-end CAD workstation that customer does not want to overclock. Ran for 72 hours and it was stable as a rock!

 

[dr/owned]

Small FFT test causes my temperatures to spike to 100C across all cores. This is with an ultra high end custom loop. First round of Blend the temps stay in the low 60s. 3rd round it's back up to 100C. After 15 minutes usually one of the threads dies, but oh well I'm not going to dial my OC back 100 Mhz to fix a use case that is never going to happen in real usage.

 

[wabbitseason]

Small FFT test causes my temperatures to spike to 100C across all cores. This is with an ultra high end custom loop. First round of Blend the temps stay in the low 60s. 3rd round it's back up to 100C. After 15 minutes usually one of the threads dies, but oh well I'm not going to dial my OC back 100 Mhz to fix a use case that is never going to happen in real usage.

Your mount is fine because the first round temperature remains low. The system hasn't reached steady-state, and I assume you've got a pretty large reservoir which is acting as a secondary heatsink until it's finally saturated. Subsequent rounds run hotter because the water in your loop is heating up, meaning your radiator or radiator fans are inadequate. How big is it? How many components is it cooling? What fans and what fan speed?

 

[sblantipodi]

Small FFT test causes my temperatures to spike to 100C across all cores. This is with an ultra high end custom loop. First round of Blend the temps stay in the low 60s. 3rd round it's back up to 100C. After 15 minutes usually one of the threads dies, but oh well I'm not going to dial my OC back 100 Mhz to fix a use case that is never going to happen in real usage.

can you post a screenshot with CPUz and prime95 running in blend mode please?
I'm really curious to see it
thanks.

 

[dr/owned]

Your mount is fine because the first round temperature remains low. The system hasn't reached steady-state, and I assume you've got a pretty large reservoir which is acting as a secondary heatsink until it's finally saturated. Subsequent rounds run hotter because the water in your loop is heating up, meaning your radiator or radiator fans are inadequate. How big is it? How many components is it cooling? What fans and what fan speed?

25 gallons, chilled water. I have temperature probes on the inlet and outlet of both my cpu and gpu loops (fully independent) so I can monitor it. I know I need a little bit more rad on the CPU loop since the temperature goes +4C over baseline. Flow rates are fine since outlet minus inlet is within 1C under full load. I have enough chiller capacity where my loop never really heatsoaks like an air cooled loop would. The only fans in the setup are on the chillers.

The strange thing Is core 1 really likes to run colder than core 4:


imageupload

 

[Deleted member 82943]

pics of the w/c loop, 100C sounds terrible!

 

[AdamK47]

People keep telling me to go for a custom loop for my 5960X. Like it's some magical cure all and something I'm supposed to be doing. I think they just like the rigmarole that goes into such a setup to set themselves apart. It's 1/3rd substance, 1/3rd placebo, and 1/3rd hip/trendy. I'll stick to my 360mm AIO cooler thank you very much.

 

[dr/owned]

People keep telling me to go for a custom loop for my 5960X. Like it's some magical cure all and something I'm supposed to be doing. I think they just like the rigmarole that goes into such a setup to set themselves apart. It's 1/3rd substance, 1/3rd placebo, and 1/3rd hip/trendy. I'll stick to my 360mm AIO cooler thank you very much.

You've just become jaded by all the pics of people's builds where it's all about color coordination, hard tubing, $5 barbs, etc. There's numerous advantages custom has over AIO at the expense of ... expense

 

[AdamK47]

People can go nuts on watercooling if that's what they like. It's just not for me. I know there are advantages. Those advantages seem over emphasised. My upgrades are on or near the release of the latest hardware. I put into perspective all the wasted effort that would have been exerted into eeking out a couple hundred MHz more. None of that effort really makes much of a difference when simply cooled next generation hardware is going to significantly outperform it anyway. The time can be used to enjoy the current generation. The money can be used for future generations.

 

[sblantipodi]

25 gallons, chilled water. I have temperature probes on the inlet and outlet of both my cpu and gpu loops (fully independent) so I can monitor it. I know I need a little bit more rad on the CPU loop since the temperature goes +4C over baseline. Flow rates are fine since outlet minus inlet is within 1C under full load. I have enough chiller capacity where my loop never really heatsoaks like an air cooled loop would. The only fans in the setup are on the chillers.

The strange thing Is core 1 really likes to run colder than core 4:


imageupload

Do you disabled thermal control?
At that temperature CPU should throttle.

 

[dr/owned]

Do you disabled thermal control?
At that temperature CPU should throttle.

Yeah I override the Tjmax in the BIOS. 89C default is weak.

 

[sblantipodi]

Yeah I override the Tjmax in the BIOS. 89C default is weak.

You are a real overclocker, nothing to add.
Raising that value on a 1000€ CPU is for addict only xD

 

[unclewebb]

Intel sets TJ Max at the factory. It is a fixed value and the correct value is written to each core of each CPU. You cannot change this value. There is a way to use an offset to decrease the thermal throttling temperature but you cannot increase this value. If you change this value in programs like Core Temp or RealTemp, these programs will not report your core temperatures correctly anymore.

Can you boot up, don't run Core Temp and then run CPU-Z and go to the About box and then click on the Save Report (.TXT) button. Have a look for this line. This shows the TEMPERATURE_TARGET register.

MSR 0x000001A2		0x00000000	0x01641000

The 64 on the right side of that register is where the TJ Max value is located. 0x64 hex equals a TJ Max value of 100°C. This data is from a 4 core mobile CPU. The 6 and 8 core desktop CPUs are probably set a little lower.

I am just curious to see what your CPU is showing.

 

[dr/owned]

Intel sets TJ Max at the factory. It is a fixed value and the correct value is written to each core of each CPU. You cannot change this value. There is a way to use an offset to decrease the thermal throttling temperature but you cannot increase this value. If you change this value in programs like Core Temp or RealTemp, these programs will not report your core temperatures correctly anymore.

Can you boot up, don't run Core Temp and then run CPU-Z and go to the About box and then click on the Save Report (.TXT) button. Have a look for this line. This shows the TEMPERATURE_TARGET register.

MSR 0x000001A2		0x00000000	0x01641000

The 64 on the right side of that register is where the TJ Max value is located. 0x64 hex equals a TJ Max value of 100°C. This data is from a 4 core mobile CPU. The 6 and 8 core desktop CPUs are probably set a little lower.

I am just curious to see what your CPU is showing.

It's adjusted in the BIOS to 120C. In CPUZ it reports 0x00781200 where 78 hex jives to 120 in decimal. Also the Intel ETU won't show throttling even after getting past 89C.

 

[Sycraft]

Mine is, went for a quite moderate OC of 4GHz. It is happy with Prime95 FMA3 tests for as long as I wish to run them. 4GHz core, 1.13 volts, 2133MHz RAM.

 

[Deleted member 82943]

102C on an 8 core chip, yikes! It hasn't fired yet?

 

[dr/owned]

102C on an 8 core chip, yikes! It hasn't fired yet?

Nope. I tried a suicide run of 1.6V and 5.1Ghz but it won't pass any stress at all. While IMO redundant to the warranty, I also have the Tuning Plan on this chip so it's like a license to throw caution to the wind and go straight for the balls out overclocks. I'd probably be running 1.4V right now if I could get the temperature under control.

 

[Deleted member 82943]

it just baffles me that an enthusiast w/c setup can't cool this beast

 

[unclewebb]

It's adjusted in the BIOS to 120C. In CPUZ it reports 0x00781200 where 78 hex jives to 120 in decimal. Also the Intel ETU won't show throttling even after getting past 89C.

In all previous Intel Core i CPUs that I have seen, TJ Max was in MSR 0x1A2 and this register was always read only so software or the bios could not change it.
Thanks for posting that info and putting your new 8 core CPU through a good work out.

 

[dr/owned]

it just baffles me that an enthusiast w/c setup can't cool this beast

There's probably a few reasons. The load - idle (downclocked) power difference on my chip is about 440W. That's a lot of heat to deal with. Second we're probably seeing the IHS limitations. I wish it was possible to delid these chips like it was with Haswell and Ivy (which I did), because at least then it would be possible to run direct die cooling which would probably make a dramatic difference in temperatures. I'd like to have the IHS either CNC'd or lapped down to expose silicon, but I don't know if the 2011 socket will be able to work with direct die cooling which requires removing the hold down brackets. The fact that 2011 has 2 brackets vs. 1 bracket on 115X would probably indicate contact pressure with the socket is a problem. Plus lapping or CNC work would probably be pretty expensive considering the tight tolerances required, and for the CNC it would require some sort of hold down fixture I suspect. Scratching the silicon wouldn't matter per say but it's only .5-.7mm thick so there's not a whole lot of meat there for error.

 

[Deleted member 82943]

yeah thats a lot to cool. still maybe a loop with 2 rads and 2 pumps could do it? i mean some of the AIO coolers are rated for 220W

 

[n=1]

There's probably a few reasons. The load - idle (downclocked) power difference on my chip is about 440W. That's a lot of heat to deal with. Second we're probably seeing the IHS limitations. I wish it was possible to delid these chips like it was with Haswell and Ivy (which I did), because at least then it would be possible to run direct die cooling which would probably make a dramatic difference in temperatures. I'd like to have the IHS either CNC'd or lapped down to expose silicon, but I don't know if the 2011 socket will be able to work with direct die cooling which requires removing the hold down brackets. The fact that 2011 has 2 brackets vs. 1 bracket on 115X would probably indicate contact pressure with the socket is a problem. Plus lapping or CNC work would probably be pretty expensive considering the tight tolerances required, and for the CNC it would require some sort of hold down fixture I suspect. Scratching the silicon wouldn't matter per say but it's only .5-.7mm thick so there's not a whole lot of meat there for error.

Careful with the IHS lapping, I remember someone saying that voids the Tuning Plan as Intel wants to see an intact IHS.

 

[dr/owned]

Sorry @OP for going off topic here.

yeah thats a lot to cool. still maybe a loop with 2 rads and 2 pumps could do it? i mean some of the AIO coolers are rated for 220W

15 minutes of running Prime using SmallFFT (max power and heat):



The top is the cpu loop, the bottom is the gpu loop. Blue/green is inlet, red/orange is outlet.

The amount of pump I'm using is adequate. I've only seen large inlet - outlet differences when I've severely restricted flow to a trickle (intentionally with a ball valve).

I need to add another rad to the CPU looking at the cpu - gpu loop difference. So doing that will probably buy me about 3C better temperatures.

I also see the GPU temperature increase about 1C with this run, so that either indicates I need more chiller capacity (work in progress) or I need more heat exchange with the chilled water reservoir (also work in progress)

Anyways, lot of words to basically say the main problem is getting the heat off the die and into the waterblock as quickly as possible. Dealing with it that heat afterwards isn't so much the issue.

 

[Deleted member 82943]

Ahh makes sense now.

 

[wabbitseason]

15 minutes of running Prime using SmallFFT (max power and heat):

The top is the cpu loop, the bottom is the gpu loop. Blue/green is inlet, red/orange is outlet.

The amount of pump I'm using is adequate. I've only seen large inlet - outlet differences when I've severely restricted flow to a trickle (intentionally with a ball valve).

snip

Anyways, lot of words to basically say the main problem is getting the heat off the die and into the waterblock as quickly as possible. Dealing with it that heat afterwards isn't so much the issue.

This is actually an interesting and unusual situation. Your first post mentioned that your CPU runs at 60C for an intial stress test, but then gets progressively hotter on stress tests immediately afterwards up to 100C (!). That sounded like a classic case of the water in your loop heating up due to an inadequate radiator setup.

Yet this post shows that the water in your loop stays under 25C. That means there's nothing wrong with your chiller and nothing wrong with your rad setup.

The mass of your processor is miniscule, and so is its heat capacity. Under constant load, it should heat up fast and reach steady-state quickly (on a minutes timescale). There's no way that the CPU is taking that long to reach steady state against water at a fixed temperature.

If there was a problem core-to-IHS or IHS-to-waterblock, it would have certainly manifested as high temps in your first stress test. Only waterblock-to-water might not show up immediately, provided you've got a big heavy waterblock that takes lots of energy to heat up.

 

[Araxie]

This is actually an interesting and unusual situation. Your first post mentioned that your CPU runs at 60C for an intial stress test, but then gets progressively hotter on stress tests immediately afterwards up to 100C (!). That sounded like a classic case of the water in your loop heating up due to an inadequate radiator setup.

Yet this post shows that the water in your loop stays under 25C. That means there's nothing wrong with your chiller and nothing wrong with your rad setup.

The mass of your processor is miniscule, and so is its heat capacity. Under constant load, it should heat up fast and reach steady-state quickly (on a minutes timescale). There's no way that the CPU is taking that long to reach steady state against water at a fixed temperature.

If there was a problem core-to-IHS or IHS-to-waterblock, it would have certainly manifested as high temps in your first stress test. Only waterblock-to-water might not show up immediately, provided you've got a big heavy waterblock that takes lots of energy to heat up.

thats how prime 95 blend test work.. its a progressive stress, then after 4th or so pass it will ramp up like if Small FFT are in use.. so all AVX instructions are in set and max FPU usage start causing instantly CPU burn... its a normal behavior with prime blend test.. when I start prime95 im in a constant 60-65c for some minutes after some passes it ramp up instantly to 78-80C. when I use small FFTs it directly go up to a steady 79-80C..

 

[PAo_ReVoLT]

Mine is only prime stable at 4.1ghz 1.21v to get to 4.2 I need 1.33v which I don't think it is worth it.

 

[dr/owned]

This is actually an interesting and unusual situation. Your first post mentioned that your CPU runs at 60C for an intial stress test, but then gets progressively hotter on stress tests immediately afterwards up to 100C (!). That sounded like a classic case of the water in your loop heating up due to an inadequate radiator setup.

Yet this post shows that the water in your loop stays under 25C. That means there's nothing wrong with your chiller and nothing wrong with your rad setup.

The mass of your processor is miniscule, and so is its heat capacity. Under constant load, it should heat up fast and reach steady-state quickly (on a minutes timescale). There's no way that the CPU is taking that long to reach steady state against water at a fixed temperature.

If there was a problem core-to-IHS or IHS-to-waterblock, it would have certainly manifested as high temps in your first stress test. Only waterblock-to-water might not show up immediately, provided you've got a big heavy waterblock that takes lots of energy to heat up.

Araxie seems to have explained it. The temperature doesn't progressively get hotter but goes through sudden jumps back and forth as the Prime95 test puts different amounts of cpu logic usage in use. Rounds 1 and 2 will be 60C, Round 3 will be 100C, Round 4 I think goes back to 70C, etc.

 

[sblantipodi]

Mine is only prime stable at 4.1ghz 1.21v to get to 4.2 I need 1.33v which I don't think it is worth it.

I would really like to know others people results on Prime95 28.x
I need 1.296V to be stable at 4.2 on 5930K

 

[Zarathustra[H]]

As title.
Asus suggest to not use the latest Prime95 small fft test with Haswell-E

Hadn't heard this.

So what they are saying now is that this generation of stock clocked chips won't even pass what used to be a requirement to consider your overclocked system fully stable?

Bullshit.

If I can't pass 24 hours of Intel Burn Test and Prime95, it's not stable, overclocked or not.

 

[sblantipodi]

Hadn't heard this.

So what they are saying now is that this generation of stock clocked chips won't even pass what used to be a requirement to consider your overclocked system fully stable?

Bullshit.

If I can't pass 24 hours of Intel Burn Test and Prime95, it's not stable, overclocked or not.

fortunantly someone who have my same idea.
Asus is widespreading everywhere that Prime95 28.x should not be used on Haswell-E since it use too much power and can degrade the CPU.

 

[Araxie]

Hadn't heard this.

So what they are saying now is that this generation of stock clocked chips won't even pass what used to be a requirement to consider your overclocked system fully stable?

Bullshit.

If I can't pass 24 hours of Intel Burn Test and Prime95, it's not stable, overclocked or not.

He its right... asus does not recommend to use Prime95 with a haswell chip any recent version.. they only recommend to use the 27.9.. 28.5 will likely fry any recent haswell chip because its not designed for such abnormal stress and can degrade abnormally fast a processor.. its posted by Asus representative in all forums, also prime95 since ivy bridge isn't considered any requirement to be fully stable...

 

[sblantipodi]

He its right... asus does not recommend to use Prime95 with a haswell chip any recent version.. they only recommend to use the 27.9.. 28.5 will likely fry any recent haswell chip because its not designed for such abnormal stress and can degrade abnormally fast a processor.. its posted by Asus representative in all forums, also prime95 since ivy bridge isn't considered any requirement to be fully stable...

people will not care buying "ultra high end motherboards" for 4.2GHz at maximum,
this is why asus is widespreading this.
Haswell-E, is not able to push clocks a lot with some decent testing (some serious bench with AVX2 like Prime95 28.5)

Most people saying about systems stable at 4.5GHz at 1.3V are using Aida at most

 

[Araxie]

people will not care buying "ultra high end motherboards" for 4.2GHz at maximum,
this is why asus is widespreading this.
Haswell-E, is not able to push clocks a lot with some decent testing (some serious bench with AVX2 like Prime95 28.5)

Most people saying about systems stable at 4.5GHz at 1.3V are using Aida at most

some people tend to forget the 30Amp recommended in the EPS12v line.. at 4.5ghz a 5960X can draw close to 25amp in the 12V EPS connector under heavy load.. they also recommend active VRM cooling above 4ghz.. Raja@ASUS made a very good guide i'll post it soon where he said all of this recommendations by Asus..

 

[Araxie]

found it.. https://docs.google.com/file/d/0Bz2VRRbLPrZnMXpJY3k5Vk8zSVU/edit

recap:

Cooling Requirements

Full-load Voltages over 1.25V fall into water-cooling territory (dual-radiator). With triple radiator water-cooling solutions, using up to 1.35Vcore is possible (depending upon ambient temps).

PSU Requirements

For overclocking 5960X processors, we recommend PSUs that can supply a minimum of 30 amps to EPS 12V. At 4.6GHz a 5960X can draw close to 25amps from the EPS12V connector under software load. Minimum recommended PSUs for Haswell-E are upwards of 1000W if using more than one high performance GPU.

VRM Heatsink Cooling

If running full load stress tests over 4GHz, active cooling of the VRM with a fan is advised due to current requirements of the 5960X processor.

Stress Testing

Users might want to avoid running Prime 28.5 small FFTs on 5960X CPUs when overclocked. Over 4.4GHz, the Prime software pulls 400W of power through the CPU. It is possible this can cause processor degradation.

 

[Zarathustra[H]]

found it.. https://docs.google.com/file/d/0Bz2VRRbLPrZnMXpJY3k5Vk8zSVU/edit

recap:

The way I read this is as follows:

Sure, you can have a fast CPU, as long as you don't actually load it all the time.

Not a big deal for most users, where CPU's spend most of their time idling, and only occasionally burst to 100% for short periods of time, but if doing serious compute-type work I'd see this as a real problem.

It's similar to your ISP promising a certain speed, but then giving you a monthly cap, or throttling you if you actually use it for more than a few minutes...

It's dishonest of CPU manufacturers to promise a speed that is not deliverable 100% of the time.

That being said, he does clarify for "overclocked" systems. If it passes all of this at stock clocks, then I have no gripe with Intel.

 

[Araxie]

The way I read this is as follows:

Sure, you can have a fast CPU, as long as you don't actually load it all the time.

Not a big deal for most users, where CPU's spend most of their time idling, and only occasionally burst to 100% for short periods of time, but if doing serious compute-type work I'd see this as a real problem.

It's similar to your ISP promising a certain speed, but then giving you a monthly cap, or throttling you if you actually use it for more than a few minutes...

It's dishonest of CPU manufacturers to promise a speed that is not deliverable 100% of the time.

That being said, he does clarify for "overclocked" systems. If it passes all of this at stock clocks, then I have no gripe with Intel.

all at stock settings its completely rock stable no matter what you do even in turbo clocks so all the advertised speed its there to work properly... however even at stock settings isn't recommended the use P95 28.5 in any haswell chip (this even include 4770K not only haswell-E) due the way P95 28.5 apply AVX2.. Aida64 3.0 and later its what manufacturers recommend for stability.. in fact when haswell was launched they warned this:

Unvalidated stress tests are not advised ( such as Prime 95 or LinX or OCCT, Intel Burn Test or other comparable applications ). For high grade CPU/IMC and System Bus testing Aida64 is recommended along with general applications usage like PC Mark 7.
Aida has an advantage as it is stability test has been designed for the Haswell architecture and test specific functions like AES, AVX and other instruction sets that prime and like synthetics do not touch. As such not only does it load the CPU 100% but will also test other parts of CPU not used under applications like Prime95.

Other applications to consider are SiSoft 2013 or Passmark BurnIn. Additionally this generation has a more specialized point of consideration for synthetic stress tests. When using an adaptive vid voltage control will be automatically controlled by the iVR when a complex concurrent AVX load is initialized from Applications like Prime95 or Aida Or LinX more voltage will be supplied than has been defined/requested.

 

[Zarathustra[H]]

all at stock settings its completely rock stable no matter what you do even in turbo clocks so all the advertised speed its there to work properly... however even at stock settings isn't recommended the use P95 28.5 in any haswell chip (this even include 4770K not only haswell-E) due the way P95 28.5 apply AVX2.. Aida64 3.0 and later its what manufacturers recommend for stability.. in fact when haswell was launched they warned this:

True, but the likes of Prime95, OCCT, LinX Intel Burn Test, etc. etc. are designed to test a worst case.

How do we know that some software/game/encoder/renderer/whatever doesn't do something similar and apply a similar load?

A CPU should be able to handle any software load thrown at it at its stock settings. if it doesn't there is simply something wrong, and the technology is being pushed too hard, and various marketing departments trying to get around inevitable Moore's Law limitations with clever hacks, limitations and marketing bs.

 

[Araxie]

True, but the likes of Prime95, OCCT, LinX Intel Burn Test, etc. etc. are designed to test a worst case.

How do we know that some software/game/encoder/renderer/whatever doesn't do something similar and apply a similar load?

A CPU should be able to handle any software load thrown at it at its stock settings. if it doesn't there is simply something wrong, and the technology is being pushed too hard, and various marketing departments trying to get around inevitable Moore's Law limitations with clever hacks, limitations and marketing bs.

all depend on how loaded are the CPU.. and how well coded are the program for that technology, remember that prime95 its 18years old.. torture test are always improved and the code of FFTs update, however its still a 18years old software not matter how much it try to implement FM3 and AVX2 it still can be unstable and produce abnormal unlimited FPU stress and voltage load (as has proven to do) .. the code in programs like aida64 was completely designed for haswell and thats why for example ASUS recommend it for system stability.. p95 its no more than a cooler testing app for haswell tech..

same as Furmark.. furmark was a good stability testing software.. today its no more than a GPU cooler/throttling testing.. =D.. some cards even at stock crash under furmark because the abnormal stress under the VRM circuitry... and even in some cases a PSU burner too... =)..