Benchmarking and forcing image quality when VRAM usage is automatically scaled

[Quartz-1]

Here it's pointed out that BF3 and the Frostbite engine automatically and invisibly scale VRAM usage. So how do benchmarkers force a certain level of image quality? Because it would give a false reading if image quality were set to Level N on two cards and one card invisibly stepped down to Level N-1.

 

[MavericK]

All that really matters is how fast the cards run the game without noticeable visual quality loss. You can nitpick about how "well it dynamically scales invisibly" but if the gamer doesn't actually see any difference then for all intents and purposes there is no difference.

 

[extide]

All that really matters is how fast the cards run the game without noticeable visual quality loss. You can nitpick about how "well it dynamically scales invisibly" but if the gamer doesn't actually see any difference then for all intents and purposes there is no difference.

I don't know if I agree with that. That kinda smells like the old 3dmark optimization drivers era...

 

[Zarathustra[H]]

I don't know if I agree with that. That kinda smells like the old 3dmark optimization drivers era...

I think it depends.

Optimization to me is just fine. The key is to give the user the best experience possible. it's a little bit like video compression. Among other things, dropping out parts that the human eye can't distinguish between in order to save storage space.

If you truly can't tell the difference without freeze-framing it, then it doesn't matter.

If - however - the changes are noticeable during gameplay, then it's a problem.

 

[Quartz-1]

All that really matters is how fast the cards run the game without noticeable visual quality loss.

I disagree. The point of benchmarking is to see how the cards run the game at a given resolution and a given level of image quality. If [H] were to review a card and say that they could run at one setting and then it were later shown that actually the game silently downgraded the graphics, then [H] would lose some of their hard-earned reputation.

 

[MavericK]

So use a different game then? I really don't understand the complaints. It's like complaining that occluded objects aren't rendered. You don't know the difference so who cares?

All games have their own way of handling VRAM and other resources. You could also say "Well, in Rage or Borderlands the textures are blurry and then get sharper so those games are invalid as benchmarks because the textures aren't loaded immediately". I don't hear a lot of complaining about that...

It's also the reason reviewers don't just use one game.

 

[Zarathustra[H]]

I disagree. The point of benchmarking is to see how the cards run the game at a given resolution and a given level of image quality. If [H] were to review a card and say that they could run at one setting and then it were later shown that actually the game silently downgraded the graphics, then [H] would lose some of their hard-earned reputation.

But if you can't tell the difference, why does it matter? IMHO, if the video card and drivers can be optimized to do more with less, then this is 100% positive.

I admit, it can be challenging to get reviews right, but when it comes to games and video cards two things matter. Apparent visual quality and frame rate. If the frame rate goes up, without a loss in apparent visual quality, then this is a win-win.

 

[Zarathustra[H]]

So use a different game then? I really don't understand the complaints. It's like complaining that occluded objects aren't rendered. You don't know the difference so who cares?

All games have their own way of handling VRAM and other resources. You could also say "Well, in Rage or Borderlands the textures are blurry and then get sharper so those games are invalid as benchmarks because the textures aren't loaded immediately". I don't hear a lot of complaining about that...

It's also the reason reviewers don't just use one game.

The comparison to z-occlusion here is very apt. Thank you for that.

 

[xoleras]

All that really matters is how fast the cards run the game without noticeable visual quality loss. You can nitpick about how "well it dynamically scales invisibly" but if the gamer doesn't actually see any difference then for all intents and purposes there is no difference.

How do you make an apples to apples comparison if one card has better LOD than another due to VRAM?

I don't care, any high end card can play bf3 fine but this has strong implications for benchmarks.

 

[socK]

I think it depends.

Optimization to me is just fine. The key is to give the user the best experience possible. it's a little bit like video compression. Among other things, dropping out parts that the human eye can't distinguish between in order to save storage space.

If you truly can't tell the difference without freeze-framing it, then it doesn't matter.

If - however - the changes are noticeable during gameplay, then it's a problem.

It really just boils down to how the game handles it.

BF3 streams textures and content in and out but does a good job of hiding it. When you revive on a completely different part of the map and your screen stays black for a second, it's streaming in content instead of just having it pop into existence before you.

UE3, Rage, GTA etc seem to opt to just try and smoothly load it into the world. It might take a second and doesn't always look great but there's no delay or stall as it's loaded. They'll just plop the bare minimum in and then go from there.


Do you try to hide the loading, just let it load in on the fly and accept the second of iffy quality, do you stall and freeze the game until everything is totally ready, etc... I'd rather have a blurry bit of content than my game grinding to a halt for a second in the middle of gameplay. BF3 just hides it, would you still have the same opinion about the engine if you got to spend a second watching content come into reality instead of a black screen that fades in and out? And it's not like you're not in control at all. You give the texture streaming a certain amount of allocated memory and it'll figure out from there how to handle it, with virtual texturing (like Rage) I'd imagine the system should only stream in what is feasible to fit on screen, so it might stream in higher quality textures at a higher screen resolution because mapping the increased density to screen makes sense. If you're using a massive texture at a lower resolution, the majority of it might just be lost data and it'll go for the lower quality version because there would be no quality loss.

 

[MavericK]

How do you make an apples to apples comparison if one card has better LOD than another due to VRAM?

I don't care, any high end card can play bf3 fine but this has strong implications for benchmarks.

But that's the thing, if you can't actually see a difference in LOD then is it really an issue?

If it's easily (or even not easily) noticeable, then sure, I'll agree. But so far I've heard zero evidence that there is any actual discernible difference.

 

[Quartz-1]

But if you can't tell the difference, why does it matter? IMHO, if the video card and drivers can be optimized to do more with less, then this is 100% positive.

But I'm not talking about the video card per se; I'm talking about the game's usage of the card. And just because I can't see it doesn't mean that someone else won't. I suppose that benchmarkers will just have to pay that much more attention to what is going on inside.

 

[xoleras]

But that's the thing, if you can't actually see a difference in LOD then is it really an issue?

If it's easily (or even not easily) noticeable, then sure, I'll agree. But so far I've heard zero evidence that there is any actual discernible difference.

It doesn't matter for actual gameplay, but if there's a method to test everything in an apples to apples fashion, wouldn't that be the best methodology?

I agree, it doesn't matter for the end user who just wants to enjoy their gaming experience. I haven't seen much evidence either, although i'd say the great majority of users have 1.5 or 1gb of VRAM.

 

[sdlvx]

It really just boils down to how the game handles it.

BF3 streams textures and content in and out but does a good job of hiding it. When you revive on a completely different part of the map and your screen stays black for a second, it's streaming in content instead of just having it pop into existence before you.

UE3, Rage, GTA etc seem to opt to just try and smoothly load it into the world. It might take a second and doesn't always look great but there's no delay or stall as it's loaded. They'll just plop the bare minimum in and then go from there.


Do you try to hide the loading, just let it load in on the fly and accept the second of iffy quality, do you stall and freeze the game until everything is totally ready, etc... I'd rather have a blurry bit of content than my game grinding to a halt for a second in the middle of gameplay. BF3 just hides it, would you still have the same opinion about the engine if you got to spend a second watching content come into reality instead of a black screen that fades in and out? And it's not like you're not in control at all. You give the texture streaming a certain amount of allocated memory and it'll figure out from there how to handle it, with virtual texturing (like Rage) I'd imagine the system should only stream in what is feasible to fit on screen, so it might stream in higher quality textures at a higher screen resolution because mapping the increased density to screen makes sense. If you're using a massive texture at a lower resolution, the majority of it might just be lost data and it'll go for the lower quality version because there would be no quality loss.

So in a situation where a card is VRAM limited (say BF3 multiplayer), the screen can randomly go black and the FPS will stay at the same rate? That's a really nice feature, specially for benchmarks. Now all it needs is some sort of GPU scaling features to jack the clocks up so the FPS can go from 100 to 150 and raise the average FPS for the game.

I guess this explains why [H] has no image quality comparison shots like they normally do and they've been replaced with a bullshit line from the NV Review Guide that basically says, "we don't know why, but the GTX 680 is a better experience because it's smoother"

 

[xoleras]

So in a situation where a card is VRAM limited (say BF3 multiplayer), the screen can randomly go black and the FPS will stay at the same rate? That's a really nice feature, specially for benchmarks. Now all it needs is some sort of GPU scaling features to jack the clocks up so the FPS can go from 100 to 150 and raise the average FPS for the game.

I guess this explains why [H] has no image quality comparison shots like they normally do and they've been replaced with a bullshit line from the NV Review Guide that basically says, "we don't know why, but the GTX 680 is a better experience because it's smoother"

Are there any intentions to post the nvidia review guide? I know the AMD review guide was posted here a while back.

 

[MavericK]

It doesn't matter for actual gameplay, but if there's a method to test everything in an apples to apples fashion, wouldn't that be the best methodology?

I agree, it doesn't matter for the end user who just wants to enjoy their gaming experience. I haven't seen much evidence either, although i'd say the great majority of users have 1.5 or 1gb of VRAM.

I think the reason why most (all?) reviewers don't simply rely on 3DMark for benchmarking numbers is because the gameplay is what matters, not just numbers.

I do get what you guys are saying but the game works the way it works. You don't have to look at the BF3 section of benchmarking if it bothers you. For people who only care about the gameplay experience in BF3 on their new cards, it works for them.

I guess this explains why [H] has no image quality comparison shots like they normally do and they've been replaced with a bullshit line from the NV Review Guide that basically says, "we don't know why, but the GTX 680 is a better experience because it's smoother"

If you have a problem with [H] testing methods you are welcome to go elsewhere.

 

[Forceman]

Someone needs to figure out a way to test whether it really does make a difference. Test with a 3GB 7970, then limit the VRAM to 2GB somehow, then test it again and see if it is faster with only 2GB. That's the only way to know if it really makes a difference.