Industry group invalidates 2,600 official Intel CPU benchmarks — SPEC says the company's compiler used unfair optimizations to boost performance

[MrGuvernment]

Probably not surprising news for anyone...

https://www.tomshardware.com/pc-com...unfair-optimizations-that-boosted-performance

SPEC says it will no longer be publishing SPEC CPU 2017 results for Intel CPUs running a specific version of the Intel compiler, citing displeasure over an apparent targeted optimization for a specific workload (via ServeTheHome and Phoronix) that essentially amounts to cheating. A note has been added to the more than 2,600 benchmark results published with the offending compiler, effectively invalidating those results, mostly from machines running 4th Gen Xeon Sapphire Rapids CPUs.

SPEC CPU 2017 is a benchmark mostly used for high-end servers, data centers, and workstations/PCs, and it tests performance in various workloads in a standardized way so that different computers can be compared to each other. Good performance in SPEC CPU 2017 hinges not just on hardware but also on software. One of the key factors in software-side optimization is the compiler, which is a program that basically takes written code and reformats it in a way that a processor can run it best.


The disclaimer that it is now attached to over 2,600 SPEC CPU 2017 results states, "The compiler used for this result was performing a compilation that specifically improves the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using a priori knowledge." This means the compiler (in this case, Intel's oneAPI DPC++/C++ Compiler) was not optimized for the kind of workload the two SPEC CPU 2017 benchmarks in question test, but specifically the two benchmarks themselves

 

[Mr. Bluntman]

Not at all surprising to me, whatsoever, due to Intel's past bad behavior.

 

[Shoganai]

 

[Zarathustra[H]]

Jeez. I thought this was something that ended with the Intel Complier lawsuit which was one of the reasons for AMD's $1B settlement with Intel ~15 years ago.

 

[sleepeeg3]

$1B was a slap on the wrist for Intel, with the untold billions in revenue they gained from cheating - why would they stop?

Unfortunately for them, they squandered their money and performance advantage. No amount of cheating will help them gain back the crown. They need to do it the old fashioned way - competing.

 

[OutOfPhase]

While not awesome, it is also a not-so-gentle reminder we should measure performance for the workloads we really run, and not attempt to extrapolate a benchmark.

 

[Zarathustra[H]]

While not awesome, it is also a not-so-gentle reminder we should measure performance for the workloads we really run, and not attempt to extrapolate a benchmark.

And that's why I miss the [H] reviews and their focus on tests that were not canned benchmarks.

 

[Randall Stephens]

I miss me some apples to apples benchmarks

 

[DukenukemX]

While not awesome, it is also a not-so-gentle reminder we should measure performance for the workloads we really run, and not attempt to extrapolate a benchmark.

It's a reminder that we shouldn't give synthetic benchmarks any credit. It's very easy to pull a VW trick where it detects it's being tested and therefore alters itself to better perform on those tests. At least SPEC realized this and removed the results.