Hey, we haven't had a nice benchmark cheating scandal in awhile, so here's one!

1

1_rick

Supreme [H]ardness
Joined
Feb 7, 2017
Messages
5,413
https://www.anandtech.com/show/15703/mobile-benchmark-cheating-mediatek

MediaTek--not phone vendors--is shipping in firmware a file that boosts CPU speeds in benchmarks. It's been detected in a bunch of phones from several different vendors, going back up to 3 years. When caught, they seem to have tried to hide what they're doing, rather than stop doing it.

What’s actually even more suspicious and we’re very lucky here in terms of catching this, is that these listings are seemingly in the process of being hidden. I had extracted the files out of my Reno3 Pro on its initial out-of-the-box firmware. Over the last few weeks OPPO had pushed a firmware update to the phone – and when at some point when I had checked something again in the file, I was surprised to see the benchmark entries disappear.


Did the mechanism get disabled? Did they stop cheating? Unfortunately, no. I don’t know where the entries have been moved to now, but the phone still very much still triggered its Sports Mode in the benchmarks with the same large performance boost. The entries weren’t merely removed, they were just hidden away somewhere else.
Click to expand...

When contacted, MediaTek said, basically, in ten times as many words, they're boosting performance to show what the CPU is capable of, without mentioning that those results won't match what you'll see using the phone.
 
1_rick said:
https://www.anandtech.com/show/15703/mobile-benchmark-cheating-mediatek

MediaTek--not phone vendors--is shipping in firmware a file that boosts CPU speeds in benchmarks. It's been detected in a bunch of phones from several different vendors, going back up to 3 years. When caught, they seem to have tried to hide what they're doing, rather than stop doing it.



When contacted, MediaTek said, basically, in ten times as many words, they're boosting performance to show what the CPU is capable of, without mentioning that those results won't match what you'll see using the phone.
Click to expand...

Hehe, just saw this on slashdot, was about to post! Good catch!
 
Just finished the article. Checking for names is just lazy. It's funny they've been doing this for years without anyone speaking up.

Remember when certain blue company was doing something much more sinister in their compiler back in the day? No worries if you don't, I haven't forgotten.
 
Armenius said:
People benchmark their phones? Don't all of them use literally the same processor with slightly varying clock speed?
Click to expand...

As said they are all very different, but like you I don't see the point. I'm not much of a phone person so if it works I don't care.

I'm also typing this on my phone. The irony is so thick you could cut it with a knife.
 
Red Falcon said:
Not even remotely, on either processor model/arch or clock speed.
Click to expand...
there can be huge differences in phone speeds now, with phone gaming becoming more and more popular it shouldn't come as a surprise. But even if two phone have the exact same specs they can have drastically different performance depending on everything right down to how well it cools itself due to under clocking from temps.
 
Armenius said:
People benchmark their phones? Don't all of them use literally the same processor with slightly varying clock speed?
Click to expand...

Phone makers benchmark their phones all over the place just like, say, GPU and CPU makers do, and for the same reason: to show them off so you'll buy them instead of someone else's.

MediaTek--not the phone makers--is doing this across at least half a dozen CPUs and phone makers.
 
dgz said:
Checking for names is just lazy.
Click to expand...

Anandtech spotted this several years ago and has been using custom binaries to prove and defeat it, so it's actually kind of sad the hardware vendors haven't caught up (but good for us).

dgz said:
It's funny they've been doing this for years without anyone speaking up.
Click to expand...

Anandtech's mentioned it before, if that's what you mean. But only, IIRC, in actual deep dives, and not in phone reviews in general (usually) so unless you read them closely it'd be easy to miss. I haven't heard it reported anywhere else.
 
THRESHIN said:
As said they are all very different, but like you I don't see the point.
Click to expand...

If you're not buying phones based on benchmark scores that are both not representative of real-world performance you yourself would see, and that if you COULD get would cause your phone to overheat and shut down, well, good on ya.

Note the article points out how that after Oppo got caught doing it, they added a "damn the torpodes mode" you could turn on--but also were honest enough to say "if you turn this on it may make your phone overheat".
 
Armenius said:
People benchmark their phones? Don't all of them use literally the same processor with slightly varying clock speed?
Click to expand...

I would say that describes the flagship models within any single generation. It doesn't really hold true once you expand to the non-flagship models or when comparing flagships across multiple generations.
 
Armenius said:
People benchmark their phones? Don't all of them use literally the same processor with slightly varying clock speed?
Click to expand...

There is a little more difference than that, but I don't get the phone nuts.

To me, pretty much all phones in the same generation and cost tier are equivalent.

A phone is a phone is a phone.

They all make calls, run messaging apps, have mobile web browsers and do email. And I don't know about you, but that's 99% of what I do with my phone.
 
Last edited:
Zarathustra[H] said:
There is a little more difference than that, but I don't get the phone nuts.

To me, pretty much all phones in the same generation and cost tier are equivalent.

A phone is a phone is a phone.

They all make calls, run messaging apps, have mobile web browsers and do email. And I don't know about you, but that's 99% of what I do with my phone.
Click to expand...
Considering mobile games generated over $60 billion last year world wide it is certainly a thing. Specs and performance are a big factor for those that do game on their phone, and like I said earlier even having the exact same specs in 2 phones can have very large differences depending on how well factors like cooling were considered in the design to allow stable clocks under load
 
Ragenrok said:
there can be huge differences in phone speeds now, with phone gaming becoming more and more popular it shouldn't come as a surprise. But even if two phone have the exact same specs they can have drastically different performance depending on everything right down to how well it cools itself due to under clocking from temps.
Click to expand...
Exactly, the cooling is also a massive factor in the consistency of performance.
Some modern smartphones have as little as 1GB of RAM, and others have 12GB+ of RAM, which can make a major difference in performance if one is simply making a phone call, or rendering 4K video content.
 
Last edited:
Red Falcon said:
Exactly, the cooling is also a massive factor in the consistency of performance.
Some modern phones have as little as 1GB of RAM, and other have 12GB+ of RAM, which can make a major difference in performance if one is simply making a phone call, or rendering 4K video content.
Click to expand...
Linus tech had a good video on phone cooling and the effects of it on processer speeds between similar chips. It's a good watch to help prove the point to
 
Zarathustra[H] said:
There is a little more difference than that, but I don't get the phone nuts.

To me, pretty much all phones in the same generation and cost tier are equivalent.

A phone is a phone is a phone.
Click to expand...
That might work for you, but there are others out there who use their phones for full-production workloads, depending on the situation.
Some people do use the phones with 8+ core CPUs and 12GB+ RAM, and will take full advantage of it, of which would not be possible on a lower-end model.

They all make calls, run messaging apps, have mobile web browsers and do email. And I don't know about you, but that's 99% of what I do with my phone.
Click to expand...
Yes, even phones dating back to the 1990s have these functions, and if that's all you need your phone for, that's great!
If someone needs more functionality in hardware for their needs, though, that is also an option, and very much shows that not all phones are equal, not even counting the differences in software/OS/hardware between Android and iOS mobile devices.
 
Ragenrok said:
Linus tech had a good video on phone cooling and the effects of it on processer speeds between similar chips. It's a good watch to help prove the point to
Click to expand...
Nice, I will have to check that out.
I've experienced it first hand, and it can definitely be a showstopper once the phone goes into cool-down mode. :p
 
Ragenrok said:
Considering mobile games generated over $60 billion last year world wide it is certainly a thing. Specs and performance are a big factor for those that do game on their phone, and like I said earlier even having the exact same specs in 2 phones can have very large differences depending on how well factors like cooling were considered in the design to allow stable clocks under load
Click to expand...

What, like Words with Friends? :p
 
Red Falcon said:
That might work for you, but there are others out there who use their phones for full-production workloads, depending on the situation.
Some people do use the phones with 8+ core CPUs and 12GB+ RAM, and will take full advantage of it, of which would not be possible on a lower-end model.


Yes, even phones dating back to the 1990s have these functions, and if that's all you need your phone for, that's great!
If someone needs more functionality in hardware for their needs, though, that is also an option, and very much shows that not all phones are equal, not even counting the differences in software/OS/hardware between Android and iOS mobile devices.
Click to expand...


Can you give some examples?
Because I can't even think of what a "full production workload" on a phone might be.

I also imagine this must be something that is exceedingly rare.
 
I've never benched my phone before -- what are the reputable Android Benchmark tools?
 
I have a very fast phone according to benchmarks. My mom's iPhone 4S still opens her email app faster. Boo.

This is like benchmarking a pen, a miserably useless metric with horrible OS's like Samsung's addon. And I really feel for people that game on their phones.
 
Zarathustra[H] said:
Can you give some examples?
Because I can't even think of what a "full production workload" on a phone might be.

I also imagine this must be something that is exceedingly rare.
Click to expand...

it's ok if other people use their phones for things you don't use your phone for.. no one should have to justify how they use their phone to you so end the crap and move on.
 
Zarathustra[H] said:
There is a little more difference than that, but I don't get the phone nuts.

To me, pretty much all phones in the same generation and cost tier are equivalent.

A phone is a phone is a phone.

They all make calls, run messaging apps, have mobile web browsers and do email. And I don't know about you, but that's 99% of what I do with my phone.
Click to expand...
My current phone has double the memory 4Gb vs 2Gb and it makes a HUGE difference :) . (still only a 100 bucks) It was a big factor in why i got this one. The phones will lag very bad when running out of usable ram:eek:
 
Thunderdolt said:
I would say that describes the flagship models within any single generation. It doesn't really hold true once you expand to the non-flagship models or when comparing flagships across multiple generations.
Click to expand...
LG uses the Snapdragon 855 in their midrange V50. Xiaomi also makes sub-$300 models using the Snapdragon 855.
 
sirmonkey1985 said:
it's ok if other people use their phones for things you don't use your phone for.. no one should have to justify how they use their phone to you so end the crap and move on.
Click to expand...

I'm not asking them to justify it. I am genuinely curious what those workloads might be, because I am being entirely honest when I say I can't picture what they could possibly be.

I'm not even aware of any app or any combination of apps that would stress even a previous gen mid range phone. Heck I still occasionally use my old LG v521 tablet from 2016 with a Snapdragon 617 and 2GB of RAM, and it is still plenty snappy.

I am inquisitive in nature. Whenever I come across anything I don't understand, I seek more information.
 
Don't know about phone benching because mine still has cans and string.
Angry Turds and Candy Crotch. That's like going back and playing Mario Bros or Sonic the Hedgehog. Current games are far more demanding graphically. Mobile gaming has been over 50% of total market share for a few years now, and only rising.
 
Zarathustra[H] said:
I'm not asking them to justify it. I am genuinely curious what those workloads might be, because I am being entirely honest when I say I can't picture what they could possibly be.

I'm not even aware of any app or any combination of apps that would stress even a previous gen mid range phone. Heck I still occasionally use my old LG v521 tablet from 2016 with a Snapdragon 617 and 2GB of RAM, and it is still plenty snappy.

I am inquisitive in nature. Whenever I come across anything I don't understand, I seek more information.
Click to expand...

Games on phones can be a demanding niche - Fortnite's probably justified a few thousand phone upgrades for gamers on the go, though I'd hate to deal with Bluetooth input and a small screen. But outside of single threaded performance for web browsing, so much of a phone's performance-demanding features are farmed out to ASICs or limited by network speed that general purpose computation's irrelevant for other day to day use. The CPUs typically have low IPC and are too constrained by cooling and power limitations to be good at number crunching. I/O also isn't the strong suit of a device that's not meant to connect to anything besides its own embedded storage unless it's through a wireless network connection. I can understand wanting to see benchmarks for a cheap desktop to understand its limitations, but outside of comparisons to other devices in the same price bracket I wouldn't waste my time.
 
Last edited:
Zarathustra[H] said:
I'm not asking them to justify it. I am genuinely curious what those workloads might be, because I am being entirely honest when I say I can't picture what they could possibly be.

I'm not even aware of any app or any combination of apps that would stress even a previous gen mid range phone. Heck I still occasionally use my old LG v521 tablet from 2016 with a Snapdragon 617 and 2GB of RAM, and it is still plenty snappy.

I am inquisitive in nature. Whenever I come across anything I don't understand, I seek more information.
Click to expand...
I’m gonna echo the response above.

I was super surprised to see my buddies 7 and 10 year old sons playing fort nite or PUBG (I cant tell the difference and was drinking with him) on an iPad and phone literally next to an Xbox one.

Honestly though especially for home use I’m surprised that they don’t just use consoles but here we are.

Blew those kids minds when I sent one my steel series controller for iOS and paired an Xbox one controller to the other kids iPhone/touch.

I’m pretty sure you can play COD either now or soon on the phone and isn’t Diablo dropping a much maligned mobile game? That’ll move handsets.

Now who the fuck is benchmarking these phones? Not me.
 
Zarathustra[H] said:
I'm not asking them to justify it. I am genuinely curious what those workloads might be, because I am being entirely honest when I say I can't picture what they could possibly be.

I'm not even aware of any app or any combination of apps that would stress even a previous gen mid range phone. Heck I still occasionally use my old LG v521 tablet from 2016 with a Snapdragon 617 and 2GB of RAM, and it is still plenty snappy.

I am inquisitive in nature. Whenever I come across anything I don't understand, I seek more information.
Click to expand...

Yeah it's either fortnite or PUBG that has a mobile port. I don't get it either. I like the fact that for the price of a flagship phone that would bench well, one could get a VERY capable gaming PC.
 
Susquehannock said:
Don't know about phone benching because mine still has cans and string.
Angry Turds and Candy Crotch. That's like going back and playing Mario Bros or Sonic the Hedgehog. Current games are far more demanding graphically. Mobile gaming has been over 50% of total market share for a few years now, and only rising.
Click to expand...

Yeah, games I can see. I didn't realize it had grown so much. I guess it is driven by low cost markets? Because I can't see anyone gaming on a mobile device by choice, if they have any other option, but maybe that's just me. The experience would have to be so bad, that I'd rather just wait until I got home to use my Desktop.

The part that made me curious was the reference to heavy production workloads above. This is the one, try as I might, I just can't come up with a mental image of what that entails.
 
1_rick said:
Anandtech spotted this several years ago and has been using custom binaries to prove and defeat it, so it's actually kind of sad the hardware vendors haven't caught up (but good for us).



Anandtech's mentioned it before, if that's what you mean. But only, IIRC, in actual deep dives, and not in phone reviews in general (usually) so unless you read them closely it'd be easy to miss. I haven't heard it reported anywhere else.
Click to expand...

Yes, I read the article. Maybe it's not that big of a deal.
 
1_rick said:
The mobile version of Farmville will melt a mid-range GPU. It's probably poorly coded, though.
Click to expand...

That is surprising. I've never played Farmville, but I thought we were essentially talking about SimCity levels of graphics.
 
You must log in or register to reply here.
Back
Top