Scott Wasson Discusses Flawed Benchmarking and New AMD Adrenalin Features

Discussion in 'HardForum Tech News' started by cageymaru, Dec 13, 2018.

  1. cageymaru

    cageymaru [H]ard as it Gets

    Messages:
    19,558
    Joined:
    Apr 10, 2003
    AMD Senior Manager of Product Management, Scott Wasson was interviewed by Gamer's Nexus where he explains why fps metrics and benchmarks are flawed. In the second video, he demonstrates some of the exciting new features like setting custom memory timings for GPUs in the AMD Adrenalin 2019 Edition 18.12.2 drivers.

    AMD's Scott Wasson walked us through the new Adrenalin driver update while the team visited GN HQ.
     
    dragonstongue and DejaWiz like this.
  2. Shagittarius

    Shagittarius n00b

    Messages:
    63
    Joined:
    May 3, 2016
    Also coming up , the Oakland Raiders talk about how touchdowns are not a good indicator of football prowess.
     
  3. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    To be fair, average framerate numbers are pretty meaningless on their own, and showing an actual plot of FPS over time actually indicates proper smoothness(or lack of in some circumstances) the way [H] does.

    That being said, it does come off as "We can't produce a high end product so please use this friendlier way to compare our products to others".
     
  4. Shagittarius

    Shagittarius n00b

    Messages:
    63
    Joined:
    May 3, 2016
    Im not saying there's no merit to either argument just that it shouldn't be coming from a team who hasn't made the playoffs for a decade. There is way more to gaming frame rates than fps, and more to football than touchdowns.
     
    Sulphademus and Armenius like this.
  5. Imhotep

    Imhotep Gawd

    Messages:
    761
    Joined:
    Feb 12, 2014
    Anybody with half a brain will tell you that framerate means nothing as long as you are in freesync range especially when you can do it with minimal frametime.. Look up the comparison that kyle,h- team have done FreeSync vs GSync.
    Comeback here with your thoughts... oh and rewatch the video because clearly you have no clue...:D
     
    geok1ng, nEo717, N4CR and 2 others like this.
  6. PeaKr

    PeaKr Gawd

    Messages:
    713
    Joined:
    Sep 6, 2004
    There is only one number I've always cared about, Minimum FPS, particularly that spot in your favorite game where FPS gets crushed by copious amounts of characters running around, complex architecture, explosions and all the eye candy maxed out. Once you've achieved that the rest is gravy.
     
    Crackinjahcs, knowom, xmadror and 6 others like this.
  7. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    Not sure if sarcasm is failing to come through due to this being text. Because regardless of what AMDs marketing tries to claim, FPS(especially MINIMUM FPS) still matters.
     
    IdiotInCharge, dgz and Armenius like this.
  8. Shagittarius

    Shagittarius n00b

    Messages:
    63
    Joined:
    May 3, 2016
    Before anyone goes crazy i want a competitive AMD, but pretending they are by manipulating data and talking down important distinctions to pretend they are competitive is wrong and helps no one, or just deceives the casual gamers. We need real competition not disingenuous praise to present the false narrative of competition when there currently is none at the high end.
     
    Armenius likes this.
  9. lenardo2

    lenardo2 n00b

    Messages:
    37
    Joined:
    Jan 7, 2016
    i can see where he is coming from when we get into the higher fps numbers though: take a 30fps increase; the higher the "base fps" is, the lower the percentage difference is for the base+30fps resultant, even though it is still a difference of 30fps

    30fps vs 60 fps = 100% increase from the "base" of 30

    100fps vs 130fps = 30% increase from the "base" of 100

    250fps vs 280fps = 12% increase from the base of 250

    etc.

    it is all relative.
    personally as long as i get a smooth frame rate(~30+) i don't care what the difference is for speed etc. but i don't do bleeding edge gaming,
     
  10. Jim Kim

    Jim Kim 2[H]4U

    Messages:
    3,374
    Joined:
    May 24, 2012
    Informative and talks about those bullshit cpu comparisons intel sponsored.
     
  11. Armenius

    Armenius I Drive Myself to the [H]ospital

    Messages:
    17,254
    Joined:
    Jan 28, 2014
    You're talking about frame time variance, and several review outlets already do this.
     
    IdiotInCharge likes this.
  12. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    Definitely. I want to buy the best performing product I can afford, AMD hasn't delivered that in well... ages. It's not that I'm an nvidia fanboy who only buys nvidia, hell even if I were I'd want them to have some competition(like AMD has finally done again in the CPU market) just so the pricing and performance isn't as much of a joke like the current 2080ti.
     
  13. Imhotep

    Imhotep Gawd

    Messages:
    761
    Joined:
    Feb 12, 2014
    Freesync vs Gsync comparison begs to dififer. Read my first sentence. Not everyone needs a 2080Ti to play in 4 K....or in some cases burn their place down...:D
     
    {NG}Fidel likes this.
  14. Derangel

    Derangel [H]ard as it Gets

    Messages:
    17,363
    Joined:
    Jan 31, 2008
    Scott has been talking about the issues of FPS for years. He was one of the people that brought frame times more into focus. He used to run Tech Report.
     
    {NG}Fidel, Sharps97, RobT and 3 others like this.
  15. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    Still no idea what the hell you're talking about since this has nothing to do with freesync or gsync, and neither of those options resolve shitty framerates.
     
    Armenius, IdiotInCharge and Jim Kim like this.
  16. bobzdar

    bobzdar [H]ard|Gawd

    Messages:
    1,616
    Joined:
    Jun 6, 2003
    Funny they show it on that laptop - I have it and Wattman doesn't work at all on it due to it being a mobile part (even though it's a full Vega56). Have to use Afterburner, unfortunately.
     
  17. Imhotep

    Imhotep Gawd

    Messages:
    761
    Joined:
    Feb 12, 2014
    Well, Then you have to do more research and rewatch the videos. Maybe you'll grasp what im talking about till then there is no point discussing it any farther. Have fun...;)
     
  18. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    Research on what? I have a gsync display, neither technology has anything to do with shitty minimum framerates.
    "hurr durr, you're wrong but I won't explain why" If you're not going to explain yourself why bother posting?
     
    Armenius, IdiotInCharge and Jim Kim like this.
  19. Imhotep

    Imhotep Gawd

    Messages:
    761
    Joined:
    Feb 12, 2014
    You are talking about highest framerates while I and the video in question talks about frametime. The Freesync vs G-sync comparison is what you need to look for...:) again, have fun !!!
     
  20. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    Can you not read? I've typed MINIMUM twice now when referring to shitty framerates. Where the hell do you get the idea that I'm talking about highest framerates? I already addressed the issue of highest FPS and average being meaningless in my first reply to this thread.

    AT NO POINT IN THIS THREAD HAVE I SAID A DAMN THING ABOUT HIGHEST FRAMERATE.
     
  21. FrgMstr

    FrgMstr Just Plain Mean Staff Member

    Messages:
    48,133
    Joined:
    May 18, 1997
    Let's all play nice and stay on topic please.
     
  22. dgz

    dgz [H]ardness Supreme

    Messages:
    5,096
    Joined:
    Feb 15, 2010
    No kidding. Imhotep is one of the most clueless users on this site. Is he just a young fool or a master puppet master that's trolling us all. Stay tuned for more!

    Edit: I saw Kyle's post too late. Watching the video now.
     
  23. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    So back to actual discussion of the topic, the guy in the video isn't wrong, it's just that he comes off as shilling for AMD(which he is). All it really boils down to is average and highest FPS not being the most relevant thing in a benchmark, giving a figure like average frametime isn't any better either since what really needs to happen is posting actual FPS or frametime(which effectively do the same thing) plots as a line to them compare so you can see where and how often the drops actually occur pulling down the minimum framerate.
     
  24. dgz

    dgz [H]ardness Supreme

    Messages:
    5,096
    Joined:
    Feb 15, 2010
    I am at minute 10 and have seen no shilling at all. He's right, and he's been saying that for years. In fact it was him (techreport) who broke the "news" to the morons a few years ago with their piece on frames and frame times.
     
    N4CR, Vader1975, {NG}Fidel and 2 others like this.
  25. Merc1138

    Merc1138 2[H]4U

    Messages:
    2,086
    Joined:
    Sep 25, 2010
    The reason I say it comes off as shilling for AMD, which it is in the sense of it's a GN video with an AMD guy in it, but it's not because the guy is an AMD employee so that's not really shilling at that point, is that regardless of his posting about frame times in the past the only reason I see for anyone from AMD to be doing that is because they still don't have anything to compete on the high end. The thing is, except for some of the worst case scenarios(usually due to the game itself), top end framerate performance does generally translate to better framerate performance on the bottom end of the chart when looking at minimum FPS. For example, I don't have a 1080ti so I can get a billion FPS, I have one so when the framerates do tank it's generally not as bad as a less performing card.

    Like I said, it's definitely not wrong, it just comes off as not being genuine when it's from a company that hasn't had anything competitive on the high end for a few years now. I would say the exact same thing if it was an nvidia rep trying to talk up maximum and avg. FPS as well.
     
    IdiotInCharge likes this.
  26. dgz

    dgz [H]ardness Supreme

    Messages:
    5,096
    Joined:
    Feb 15, 2010
    I feel the opposite. I salute honest talks from company representatives. So what if an nvidia engineer is trying to talk up maximum and average FPS. I am all for people hyping high refresh rates. I want 1KHz movies and games
     
  27. IdiotInCharge

    IdiotInCharge [H]ardForum Junkie

    Messages:
    9,795
    Joined:
    Jun 13, 2003
    The problem with talking about 'frames per second', be they maximum, average, minimum... is that they are measured over the course of a second. In one second, you could have one 500ms frame and sixty 8.3ms frames, and you'd have 61FPS- and it'd look more like 2FPS. Hard stuttering if it repeats.

    This is why frametime analysis is a big deal. Frametimes are what we've measured to generate FPS numbers, but actually looking at them tells us how gameplay actually feels.
     
  28. Gideon

    Gideon 2[H]4U

    Messages:
    2,061
    Joined:
    Apr 13, 2006
    I just want to point out there is a whole other market that matters as well. AMD may not have the highest end card but they have great cards for the rest of the market. I also have seen many sites that like to point out max fps like it means something, part of what started the canned benchmark craze. Stuff it all in a buffer and watch those max frames soar, but it's nothing like how the game will play. Part of the reason I like the video card reviews here is they actually play the game like I would. Nothing wrong with reminding people to look at a bigger picture then just 1 number on a chart.
     
  29. Evil Scooter

    Evil Scooter [H]ard|Gawd

    Messages:
    1,896
    Joined:
    Sep 22, 2000
    It'd be a lot better if AMD would just make better products to compete with Nvidia on the high end. last time that happened was with the 7970's and I ran 2 in xfire. Make a better product and you will not need to expend the cycles spinning how blue is the new red.
     
  30. I understand why frame time is important. I also understand the point in the video about the difference between 200 and 230 FPS being of less meaning. However, I have not had the money in my life to build many systems that run a 2K monitor 230+ FPS in the games I play. What I'm most interested in is how a card runs games I want to play at 2K because I have a 2K monitor.

    I agree with what Merc says about minimums being more telling than maximums. If a game can't maintain greater than 30FPS I don't want it even if it maintains consistent frametimes and doesn't tear.

    I also think it's worth noting that system configuration, game code, and many many other factors contribute to both FPS and Frametimes. Vanilla Skyrim probably still stutters on a 2080ti even though the FPS is fine. I have seen too many posts from people complaining about "(insert video card here) sucks" and then how they are "running a top of the line (insert shitty CPU here)" with a brand new (insert 5400 RPM shitty drive here)".

    If frame rates or frame times get you excited, great!

    I think the Kyle and his crew have the right of it. The best testing is real world experience testing. How does my game or a similar game play on the card. Is it smooth or does it suck?
     
  31. next-Jin

    next-Jin [H]ardness Supreme

    Messages:
    6,137
    Joined:
    Mar 29, 2006
    I think everything changed for me once I got a pair of Crossfired 7970s. I was running games over 60fps at all times yet somehow it felt like shit.

    Low and behold the whole frame time measurements started to become huge and everyone realized just how important minimum frame rates and frame times are.

    The only way I was ever able to make Crossfire feel alright was to set my max frame limit to 59 with no syncing going on. If I enabled v sync the input lag was horrible.
     
  32. Kor

    Kor 2[H]4U

    Messages:
    2,176
    Joined:
    Mar 31, 2010
    Trying to sort out frame times on 5870 E6 Crossfire is what finally drove me to setting up a [H] forum account after lurking for the better part of a decade :D
     
    Jim Kim likes this.
  33. ph0ng

    ph0ng n00b

    Messages:
    38
    Joined:
    Feb 9, 2017
    He wasn't saying that at all. He was just trying to explain why frame-time is a better metric for analysis, and that frame-time isn't linear. 30 to 60 fps is a huge leap in frame-time, but you get diminishing returns with each doubling of frame-rate in theoretical frame-time. Also, real-world frame-time on real-world hardware starts to suffer and have these hiccups that can't be seen on an average FPS graph.

    Going from 144FPS to 300FPS is still kinda noticeable, but it's much less so than any previous doubling, which is surprising as it's computationally hugely more expensive. This is because the frame-time is less significantly effected. Like tenths of 1ms.

    That's all the guy was saying. I understand the host kept wanting to drum up some controversy and steer the conversation, but the guy wasn't taking the bait, and sincerely wanted to explain this fact to people.