AMD Slams SYSmark Benchmark, Calling It Biased And Unreliable

Megalith

Megalith

24-bit/48kHz
Staff member
Joined
Aug 20, 2006
Messages
13,000
AMD has released a video with their thoughts on this popular benchmark.

In AMD's testing SYSmark AMD has found that SYSmark provides data that is not comparable to the difference in real world performance when it compared AMD and Intel CPUs, giving a performance delta of 50% for a comparable system when other benchmarks like PCMARK only give a delta of 7%.
 
only under preforming CPU i ever got was AMD's top 4 core
 
I haven't watched the video, but I always felt it's weird that AMD has had in the past several times publically admitted bias or disagreement with stuff (How the treated Kyle with the Nano situation).

It's one thing for some rag or third party site to do it, but when an OEM does it it's just odd to me.

Like why spend money that could be used for something else to make this video.

Hell this video was probably edited by an Intel :)
 
watched it while I wrote the above post.

What is this shit

AMD Dude : "Well the best way to test this is to have a third party do it, so we at AMD internally created some scripts to test the Intel and AMD processors".
 
I'm all for AMD but this is bullshit. SYSMark is doing it correctly by measuring the CPU, cause that's what it does. Now if SYSMark wasn't using certain instructions because not Intel and therefore no AVX or SSE3 instructions will be used, thus hurting AMD results.

This sounds like since AMD uses good GPUs in their systems that the benchmark isn't using the GPU, which honestly most applications don't. And AMD is showing laptops which makes sense to have a good GPU because that's all you're getting with that laptop. With a desktop the included GPU on the CPU doesn't matter because everyone buys a graphics card. A $100 graphics card is faster than what's included with the CPU.

Now if DX12/Vulkan allows us to combine GPUs to add more processing power then a good APU makes sense and AMD have a valid case. But that isn't the case at the moment. So yea SYSMark would be a correct representation of how applications work today.
 
AMD have some very good points here.

Interesting to see what will happen.
 
Basically they're trying to say that Sysmark (who uses this?) doesn't measure the iGPU and add it in to artificially inflate the scores to make their shit mobile CPU's look better.

What this really does is make me nervous about Zen if they're dredging this stuff back up at this point in the game.
 
Synthetic benchmarks are only useful as a paid marketing tool for the product manufacturer but useless from a consumer point of view.
 
Well no shit, fortunately real world testing is really easy. It's time consuming, but it's easy.

AMD's last generation of processors managed to be pretty inferior to Intel in both, though.

If only there was a web site that did real world testing *cough* [H]
 
Throw Prime95 or x264 encoding at the chip with StaxRip and see whats faster then

"Benchmark" suits suck because they never test real world performance, just some weird "How fast can you open Microsoft Edge" etc thing
 
So is it another episode of benchmark X sux because it doesn't use our glorious igpu ?
 
All this time, I was just waiting for them to push the big red button. Motherf.......s
 
The grammar could have been more clear.

The 3rd party referred to was Microsoft. In that Office was used to do real world testing.

I wonder if those scripts have been made public? That would be transparent about the lack of transparency.

If GPU resources are used in word then why are they not also used in just about every other real world application? If they are not used in word what is the point of saying that having an integrated GPU is useless?

It is clear that every poster so far has bias in this matter. To many fanboyisms.

To be clear I am not excluding myself from have some bias - who can go though this life of ours and not have at least a little?
 
AMD keeps finding to ways to motivate Intel to bury them even further into the ground than they already are :D
 
Differnt bencmhark measrue differnet part of the cpu's ?

When AMD is selling thier top of the line cpu with only ondia GPU benchmarks og intergers (7-zip) it hard to see them calling other benchmarolsfor biased. since the totally sidestepped their bad FPU performance in their own marketing benchmarks.
 
SvenBent said:
Differnt bencmhark measrue differnet part of the cpu's ?

When AMD is selling thier top of the line cpu with only ondia GPU benchmarks og intergers (7-zip) it hard to see them calling other benchmarolsfor biased. since the totally sidestepped their bad FPU performance in their own marketing benchmarks.
Click to expand...

You might want to buy a new keyboard or rearrange keys on your current one.:)
 
Out of curiosity, since people mentioned this is iGPU based also. Would it be wrong to want to see a Broad i7-5775 used in this comparison towards one of AMD's chips?
 
trentchau said:
Out of curiosity, since people mentioned this is iGPU based also. Would it be wrong to want to see a Broad i7-5775 used in this comparison towards one of AMD's chips?
Click to expand...

Then AMD would just bitch that the price brackets are different :p
 
Step 1: Rage quit BAPCo
Step 2: Whine for 5 years
Step 3: ...
Step 4: Fade into irrelevance
 
People on here miss the point when you use a benchmark it has to have some value to it.

That point is gone if Intel pays/cuddles/strokes Bapco to not include optimized code for any competitors.

The same would be valid if you have a 3Dnow! benchmark and tell everyone it sucks on Intel because it does not support the instruction set or really badly.

Value or information can only be valid if the test is valid, Via which holds x86 license as well stopped with Bapco as well...
 
Pieter3dnow said:
That point is gone if Intel pays/cuddles/strokes Bapco to not include optimized code for any competitors.
Click to expand...
That's typical of the ignorance behind most complaints about Sysmark.

AMD complains about 2 things: the weighting of different sub-tests and feels Sysmark should include more GPGPU tests. While AMD was at BAPCo, it approved of most of the milestones (80%) while involved with developing Sysmark before it rage quit.

The applications in the Sysmark suite and not "optimized for Intel" anymore than they're optimized for AMD. They're standard, widely used commercial applications. These are the versions used in Sysmark 2014, which is getting a bit old:

Adobe® Acrobat® Pro XI
Adobe® Photoshop® CS6 Extended
Adobe® Premiere® Pro CS6 (On 64-bit Operating Systems)
Adobe® Premiere® Pro CS4 (On 32-bit Operating Systems)
Google® Chrome® 32.0
Microsoft® Excel® 2013
Microsoft® OneNote® 2013
Microsoft® Outlook® 2013
Microsoft® PowerPoint® 2013
Microsoft® Word® 2013
Trimble® SketchUp™ Pro 2013
WinZip® Pro 17.5
WinZip® Command Line 3.2
Click to expand...

Almost all of those use the MS C++ compiler (very vendor neutral), and NONE use the Intel compiler.

The reason AMD sucks at Sysmark is because its processors suck at virtually all benchmarks because they have terrible IPC. In the video, AMD compares an old dual core i5 to a quad core FX mobile processor. Even in the benchmarks AMD uses to demonstrate it "doesn't" suck as badly as Sysmark suggests, it loses in both.

We've all seen a wide range of benchmarks run on AMD processors, so unless you're completely delusional, you know the problem isn't Sysmark.
 
If benchmarks matter so much, then it would be in your best interests to design chips that score well on the benchmarks.

If Benchmarks don't really matter at all, Don't give them more credibility be complaining about them.

Essentially, AMD is doing the worst possible thing here.
 
pxc said:
That's typical of the ignorance behind most complaints about Sysmark.

AMD complains about 2 things: the weighting of different sub-tests and feels Sysmark should include more GPGPU tests. While AMD was at BAPCo, it approved of most of the milestones (80%) while involved with developing Sysmark before it rage quit.

The applications in the Sysmark suite and not "optimized for Intel" anymore than they're optimized for AMD. They're standard, widely used commercial applications. These are the versions used in Sysmark 2014, which is getting a bit old:



Almost all of those use the MS C++ compiler (very vendor neutral), and NONE use the Intel compiler.

The reason AMD sucks at Sysmark is because its processors suck at virtually all benchmarks because they have terrible IPC. In the video, AMD compares an old dual core i5 to a quad core FX mobile processor. Even in the benchmarks AMD uses to demonstrate it "doesn't" suck as badly as Sysmark suggests, it loses in both.

We've all seen a wide range of benchmarks run on AMD processors, so unless you're completely delusional, you know the problem isn't Sysmark.
Click to expand...

Except the complaint is that they specifically cherry pick individual tasks those programs perform to favor Intel. The complaint isn't just a recent complain either. It happened before when the Athlon XP came out and beat the Pentium 4 in Sysmark 2000, only for Sysmark 2001 to be released where again the Pentium 4 outranked the Athlon XP.

The two processors AMD is comparing in that video is the AMD FX-8800P and i5-5200U. No, the i5-5200U is not 50% more powerful than the FX-8800P. Passmark ranks the AMD above the Intel
 
variant said:
Except the complaint is that they specifically cherry pick individual tasks those programs perform to favor Intel. The complaint isn't just a recent complain either. It happened before when the Athlon XP came out and beat the Pentium 4 in Sysmark 2000, only for Sysmark 2001 to be released where again the Pentium 4 outranked the Athlon XP.
Click to expand...
No, that's just from a bad memory and XP performance inflation. It happens all the time.

If you remember, the advantages the XP had over P4 was a higher IPC. Back then AMD was still using performance ratings. For example, the Athlon XP 1900+ ran at 1.7GHz. The observation back then was it took an "X"MHz faster P4 to match a lower clocked XP processor. There was no magic there. Pentium 4 chips running 900MHz faster than the fastest AXP were available when Sysmark 2000 was out.

The Athlon 64 with its on-die memory controller was a different story. The performance gap was widened significantly, to the point where a 1GHz+ lower clocked A64 could outperform P4 chips. People seem to confuse the two situations.

For example, this XP benchmark compared against P4 models: http://www.tomshardware.com/reviews/athlon-xp-2100,436-15.html

Depending on the model, XP chips running at 300MHz lower clock speeds were keeping up with the P4 in Sysmark 2000 and Sysmark 2002. Is this an outlier? No. The same pattern is seen in most of the other benchmarks. AMD had a slightly better edge in FPU heavy benchmarks like Cinema 4D and LAME, but only won in one. Roughly, you could say that it took a Northwood Pentium 4 with 300-500MHz higher clock speed to beat an AXP.
 
DarthBeavis said:
AMD sucks
Click to expand...

How droll.

Even the most blinded Intel fanboys must understand that, by merely existing, AMD is keeping Intel's prices in check. You don't have to support or even like AMD, but if anyone should be cheering on Zen it'd be the Intel supporters - if Zen fails and AMD goes down once and for all, prepare your anus for some sweet price hikes.

variant said:
Except the complaint is that they specifically cherry pick individual tasks those programs perform to favor Intel. The complaint isn't just a recent complain either. It happened before when the Athlon XP came out and beat the Pentium 4 in Sysmark 2000, only for Sysmark 2001 to be released where again the Pentium 4 outranked the Athlon XP.

The two processors AMD is comparing in that video is the AMD FX-8800P and i5-5200U. No, the i5-5200U is not 50% more powerful than the FX-8800P. Passmark ranks the AMD above the Intel
Click to expand...

Intel initially passed on FMAx instructions, which AMD has been implementing since Bulldozer as FMA3 and FMA4 (Intel did later implement FMA3). These instructions would have helped shore up AMD's lacking FP/IPC performance if devs had implemented in their software...but since the giant didn't see fit to bless it, AMD (and the rest of us) suffer. This is the sort of above-board anticompetitive behavior that Intel deals in, and if you look below the table you will see lots of examples where they tried to sabotage AMD.

It's all business ethics-type stuff, but if these Intel fanboys think Intel loves them back they've got another thing coming.
 
pxc said:
No, that's just from a bad memory and XP performance inflation. It happens all the time.

If you remember, the advantages the XP had over P4 was a higher IPC. Back then AMD was still using performance ratings. For example, the Athlon XP 1900+ ran at 1.7GHz. The observation back then was it took an "X"MHz faster P4 to match a lower clocked XP processor. There was no magic there. Pentium 4 chips running 900MHz faster than the fastest AXP were available when Sysmark 2000 was out.

The Athlon 64 with its on-die memory controller was a different story. The performance gap was widened significantly, to the point where a 1GHz+ lower clocked A64 could outperform P4 chips. People seem to confuse the two situations.

For example, this XP benchmark compared against P4 models: http://www.tomshardware.com/reviews/athlon-xp-2100,436-15.html

Depending on the model, XP chips running at 300MHz lower clock speeds were keeping up with the P4 in Sysmark 2000 and Sysmark 2002. Is this an outlier? No. The same pattern is seen in most of the other benchmarks. AMD had a slightly better edge in FPU heavy benchmarks like Cinema 4D and LAME, but only won in one. Roughly, you could say that it took a Northwood Pentium 4 with 300-500MHz higher clock speed to beat an AXP.
Click to expand...

Why did you just link Sysmark 2002 benchmarks? That in no way defeats the claim that Athlon XP beat Pentium 4 in Sysmark 2000 and then the whole thing flipped later.

Sysmark 2000
iwdP4sA.jpg


Not only did the Athlon XP beat the Pentium 4, it soundly beat the Pentium 4.
 
buzzbomb said:
Intel initially passed on FMAx instructions, which AMD has been implementing since Bulldozer as FMA3 and FMA4 (Intel did later implement FMA3).
Click to expand...
No, that's not what happened. Wikipedia has an overview of what did happen: https://en.wikipedia.org/wiki/FMA_instruction_set#History

Briefly, AMD was creating a separate SSE5 extention, which contained FMA3. Intel was already working on AVX, which is based on 3 operand instructions, but early specs had a 4 operand FMA instruction. Later that year, Intel changed the spec to use 3 operand FMA. A few months later AMD changed its spec to 4 operand FMA based on the 13 month old Intel FMA spec instead of the 6 month old spec revision. Two years after the last AVX spec revision, Intel released Sandy Bridge with AVX, but without the FMA3 instruction in the AVX spec. I think that was intentional.

It was just an unusually messy process.
 
variant said:
Why did you just link Sysmark 2002 benchmarks?
Click to expand...
Oops, I thought I linked to Sysmark 2000 benchmarks in that link. It must have been in another tab I had open.

But that image you posted is the Willamette P4, not the Northwood P4 in the other comparison. It's not directly comparable. Worse, it's from that nut Van who was consumed by his own non-reproducible testing (booted from THJ) and his own conspiracy mindedness.

Here is Sysmark 2001 showing the same results as Sysmark 2002:
http://www.xbitlabs.com/articles/cpu/display/northwood-2200/
http://www.anandtech.com/show/866/5

Here are some more Sysmark 2000 results showing the same pattern in Sysmark 2001 and 2002:
http://www.chip.de/artikel/CPU-Duell-Pentium-4-Northwood-gegen-Athlon-XP-2000-2_12854515.html

There was a problem in Sysmark 2001, where Windows Media Encoded checked the CPU ID string to enable some optimizations. See here for the problem: http://www.realworldtech.com/fair-benchmarks/ That is not a problem in Sysmark 2000 or 2002 though, and only affected the multi-media sub-score.
 
variant said:
Also, people should remember that BAPCo used to actually reside at Intel's headquarters and headed by an Intel employee. The claim has always been that BAPCo was nothing more than a front for Intel.

http://www.theregister.co.uk/2000/07/24/intel_and_bapco_just_good/
https://www.linkedin.com/in/john-peterson-390b39a8
Click to expand...
So was PCI SIG, and pretty much every other standard before it was spun off to different standards groups. If you need to dig back almost 16 years to prop up a conspiracy, you're getting pretty desperate.

Not even AMD claimed that was a problem when it rage quite BAPCo in 2011 or in the latest crybaby video it released. lol
 
A sample of how the Sysmark scores just kept getting worse for AMD with each new version of it.

Sysmark 2000 Internet Content Creation
Athlon XP 2000: 341
Pentium 4 2GHz: 263

Sysmark 2001 Internet Content Creation:
Athlon XP 2000: 203
Pentium 4 2GHz: 212

Sysmark 2002 Internet Content Creation
Athlon XP 2000: 215
Pentium 4 2GHz: 238
 
Was the operating system the same throughout those years? Was the hardware identical? Same ram, same GPU's..

I could use the same CPU on 85 tests and get different results each time if the systems weren't identical.

Not just that, but Windows XP was out for the 2001 test but not the others. That right there would be a huge change in scores.
 
To add to that.. maybe a newer OS or ram type allowed the Intel CPU to pull ahead? All things we have no idea about.

Without more info its all guess work.
 
pxc said:
So was PCI SIG, and pretty much every other standard before it was spun off to different standards groups. If you need to dig back almost 16 years to prop up a conspiracy, you're getting pretty desperate.

Not even AMD claimed that was a problem when it rage quite BAPCo in 2011 or in the latest crybaby video it released. lol
Click to expand...

Are you seriously trying to compare a "benchmark development consortium" to a standard? It's an organization that releases a benchmark tool, that not surprisingly shows Intel in the best light.

Don't you mean when Nvidia, AMD, and VIA all left BAPCo at the same time in 2011?
 
You must log in or register to reply here.
Back
Top