How Accurate are Canned PC Benchmarks?

Discussion in 'HardForum Tech News' started by rgMekanic, Apr 23, 2018.

  1. rgMekanic

    rgMekanic [H]ard|News Staff Member

    Messages:
    3,897
    Joined:
    May 13, 2013
    DigitalFoundry has posted a new video on YouTube, asking the question "Just How Useful/Accurate Are PC Benchmarks Anyway?"In the video Alex goes over the differences between canned benchmarks like those found in Tomb Raider, and Metro Last Light to see how they compare with actual in game performance.

    This is quite a problem, where canned benchmarks do not necessarily represent actual performance while playing the title. I can't believe it's taken someone so long as to think that canned benchmarks may misrepresent gameplay. Oh wait... Kyle first pointed it out in 2001 with the Quake 3 Arena benchmark.... And again in 2003 Kyle and our GPU Editor Brent Justice looked at the validity of 3DMark03... Culminating in Kyle and Brent working together once again in "Benchmarking the Benchmarks" in 2008 when they completely dissected real world gameplay vs Timedemo benchmarking in Crysis.

    Check out the video

    Anyone want to bet if Digital Foundry keeps using built-in benchmarks. I won't, but hey, maybe it will make more people aware of the problems with built-in demos, and hopefully stop giving them so much weight when predicting video card performance. But with anything in the PC or PC gaming world, don't forget. Thanks to cageymaru for the story.
     
  2. Simmonz

    Simmonz 2[H]4U

    Messages:
    2,506
    Joined:
    May 14, 2008
  3. Azphira

    Azphira [H]ard|Gawd

    Messages:
    1,822
    Joined:
    Aug 18, 2003
    I miss the old benchmark.

    ~
    timedemo 1
    demo four


    But it doesn't work well anymore since the game caps at 1000fps now.


    Where was this 1000fps back when I actually NEEDED it?
     
  4. Grimham

    Grimham [H]ard|Gawd

    Messages:
    1,427
    Joined:
    Jul 20, 2004
    Canned benchmarks are still very useful for tuning cards, mods and such. You need a reproducible benchmark.
     
    craigdt and dragonstongue like this.
  5. Stoly

    Stoly [H]ardness Supreme

    Messages:
    6,335
    Joined:
    Jul 26, 2005
    I stopped using benchmarks decades ago. I only used 3dmark vantage because I got a free key and then only to watch the demos.

    Actually, I tried 3dmark 2000 and 2001 a few months ago just for kicks.
     
  6. trick0502

    trick0502 [H]ardness Supreme

    Messages:
    5,141
    Joined:
    Apr 17, 2006
    Sites use canned benchmarks because they are easy. I’m sure Kyle and Brent would agree with that.
     
    FrgMstr likes this.
  7. IKV1476

    IKV1476 Lurker

    Messages:
    283
    Joined:
    Dec 26, 2005
    I remember when Kyle started writing about benchmarking and ultimately playing the games to find the true performance all those years ago. I was an angry pc gamer that spent years trying to figure out why I could run a time demo and get great fps but then when I played the game I had to turn down settings and still not get the fps that a time demo gave me. When Kyle flipped the benchmark world on its head, I finally knew what was going on.
    I used to hit a dozen website's for benchmarks with no one in particular first, but when HardOCP started real world benchmarks this was the first place I came to get the information I needed. And most of the other sites I just eventually stopped going, they just kept them canned mostly useless benchmarks.

    Now I live here.
     
  8. WhoMe

    WhoMe Gawd

    Messages:
    827
    Joined:
    Jan 3, 2018
    About as accurate as a can of spam tastes like a home cooked ham?
     
    Revdarian, FrgMstr, Simmonz and 3 others like this.
  9. IKV1476

    IKV1476 Lurker

    Messages:
    283
    Joined:
    Dec 26, 2005
    Live and learn. Every video card review back then pounded canned down our throats. No one said anything about performance other than canned. Of course I was aware that my games ran worse when playing, but no one seemed to talk about real game play in reviews. I didn't visit forums much back then. I read articles and played games. It is what it was for me.
     
    rgMekanic likes this.
  10. lostin3d

    lostin3d [H]ard|Gawd

    Messages:
    2,040
    Joined:
    Oct 13, 2016
    I'll use them initially when I 1st fire up a game or make some significant hardware or setting changes but generally speaking I'll try to keep a save on hand where I notice pretty major drops in fps. Generally speaking though, canned sucks in terms of accuracy it should only be considered a generic baseline but by no means the definitive gauge or needs of a game.
     
    bboynitrous likes this.
  11. cocheese256

    cocheese256 Gawd

    Messages:
    784
    Joined:
    Apr 23, 2005
    I install a new game, the first place I go are the options before I even start playing the game. I look at the auto detect settings and then hit play. Play for a few minutes. If game looks good and plays well, I play. If it plays really smooth I go back and bump up some settings until I either max out or the game reaches my thresh hold of play-ability.
    I like to read the reviews here(both official and on the forum) to get an idea of what I SHOULD be getting performance wise. It is a good review process (probably the best), but it is not the bible. What my machine does is what my machine does.
     
  12. katanaD

    katanaD [H]ard|Gawd

    Messages:
    1,987
    Joined:
    Nov 15, 2016
    the only benchmark i follow nowadays is a really simple one.. am i having fun?
     
  13. Putz

    Putz I have a custom title

    Messages:
    5,281
    Joined:
    Jul 8, 2002
    the only thing i find canned benchmarks good for is comparing one card directly to another to get a consistent idea of how they compare to one another, but i don't think they transition to real world with a pinch of poo

    also good for testing overclocks

    bout all
     
  14. ole-m

    ole-m Limp Gawd

    Messages:
    446
    Joined:
    Oct 5, 2015
    I3 @ 5.5 ghz will outperform my Ryzen 1700 in benchmarks for games, I know which I like living with.
    The one where a youtube playlist of music can be playing in background, installation of a steam game, my 70 other chrome tabs, discord voice chat in background.
    maybe some coding applications are running in background.

    I've never closed anything on my pc since my 1700 due to performance related issues, I just load it and use it.

    a mate with i5 7600K is already having issues with closing background tasks, funny thing is that two mates have near identical systems; 7700K vs 7600K is the only difference cause one of them looked at gaming benchmarks ONLY.
    he's planning on upgrading already

    This is however impossible to benchmark, but it's something we all do to some extent.
     
    Revdarian likes this.
  15. JackNSally

    JackNSally Limp Gawd

    Messages:
    151
    Joined:
    Dec 7, 2011
    I'm pretty much the same. Am I having fun and the game running better than a PowerPoint presentation? (Roughly 45 FPS+)
     
  16. Sparky

    Sparky 2[H]4U

    Messages:
    3,202
    Joined:
    Mar 9, 2000
    who else bought Quake 3 Arena for the benchmark then fell in love with the game?
     
  17. zkostik

    zkostik Gawd

    Messages:
    931
    Joined:
    Sep 17, 2009
    Yeah, canned benchmarks are bullshit if you want to tell game performance and have been so for a VERY long time. I only use 3dmark as part of stress testing as it can load processor as well as gpu/s. It's good for SLI stress testing as well as determining full load temps but in gaming world it's unlikely one will ever see full load across the board. My machines are dual use so gaming and workstation so I do find stress testing being useful, otherwise real game frame time is the way to go. Regarding your friend having to close background tasks, it isn't processor issue but something with his software or he built wrong machine for his use. He probably need to learn more about hardware and software more than upgrading his machine (otherwise it sounds almost like people who are buying a new computer when they pick up malware and it slows down and then their new computer gets same shit and slows down as well) I sometimes have hundreds of tabs and browser taking over 10GB of RAM with no issues. Pretty much only reboot for Windows or other big updates.
     
  18. mnewxcv

    mnewxcv [H]ardness Supreme

    Messages:
    6,463
    Joined:
    Mar 4, 2007
    I liked how some games had in game benchmarks. I know some still do, but that really helped tuning the in game settings to where I want them. I like how forza7 has one since there are so many video settings.
     
  19. ole-m

    ole-m Limp Gawd

    Messages:
    446
    Joined:
    Oct 5, 2015
    he looked at gaming benchmarks,
    I tried to convince him to buy a 7700K instead but he said no difference.
    he regrets...

    I try to inform that more cores than required for target task is always good.
    I installed both systems and I know that is the only difference, it didn't take long before that difference was noticed.

    They are both simple users, but simple users browse, chat on discord, play music through spotify... the general stuff but still amazing what just HT alone can do.
     
  20. M76

    M76 [H]ardForum Junkie

    Messages:
    9,338
    Joined:
    Jun 12, 2012
    I thought it was common knowledge that canned benchmarks are for comparing the performance of different hardware under the same circumstances only. And they're not indicative of the game's actual real world performance.
     
  21. Hornet

    Hornet [H]ardness Supreme

    Messages:
    6,608
    Joined:
    Oct 4, 2005
    I use benchmark when overclocking my graphic card, as tools like 3DMark are still some of the more stressful applications around.
    And as other mention, they provide a consistent means of comparison when you're tuning your hardware.

    Not very useful for real gaming experience, but if you enjoy overclocking your hardware just for the fun of it, these tools are probably more useful than actual games.