AMD vs Nvidia Encoders

dave343

[H]ard|Gawd
Joined
Oct 17, 2000
Messages
1,810
I've been using the Nvenc encoder for a while now and it's been pretty decent, but with the launch of the new Navi 2 cards, I'm interested in switching over but wanted to test the AMD encoder out first. With my Nvidia card, I've been using the following settings to record gameplay; Nvenc h.264, 1440p canvas and output, @60fps, CQP 15, and Maximum Quality. Usually for a 30-40min capture it dumps out a roughly 10GB file, recorded in MKV which I then run through Handbrake on almost placebo quality to get a MP4 which I then edit in Resolve 16 (free). I recently tried bypassing handbrake and just recording straight to MP4, but a 50min capture was 40GB. Not that I have an issue with the file size but that's quite a lot bigger.

Anyways... to test the AMD encoder, I tossed a 5700xt into my rig, and used the exact same settings, except obviously choosing the AMD h.265 encoder (as I hear AMD's h.264 is bad) and I also recorded in 1440p, 60fps, and CQP set to 15. The results I'm getting are not good... and I'm looking to find out if there is anything I can do. The primary issue I'm finding is that recording with Nvenc, Nvidia seems to retain much, MUCH more retail, and this is especially apparent in dark corner areas. With the AMD encoder, it seems to give extreme color banding? (if that's what it's called), in the darker areas. In outside enviroments, the detail seems almost the same, but since I've been playing the new Amnesia Rebirth game, I've been noticing the quality loss in dark area's when using AMD's h.265 encoder.

Also, fwiw, I did try recording using h.264 off the CPU, but even with faster preset in OBS, it's laggy as hell. Anything less, like fast, or medium will make things a power point presentation in the video playback.

So, here is a comparison shot between a dark area in Amnesia Rebirth, the leftside is Nvidia, and the rightside AMD. I know a picture doesn't do things justice but you can still see how Nvidia retains a lot more detail in dark area's. This isn't to say Nvidia's Nvenc is fault free, because I still notice color banding, but it seems to be far, far less, and the picture seems sharper with more detail. If you look at the floor you can see how Nvidia seems to retain detail where with the AMD recording it's really bad, and I'm hoping there are settings or things I can try to improve it to match Nvidia's nvenc.

amd_nv_encoding.png


And here's a 2nd shot of just AMD. You can see on the floor, close the barricaded furniture those dark areas look bad.

1604424334485.png
 
Last edited:

chameleoneel

2[H]4U
Joined
Aug 15, 2005
Messages
3,718
I've been using the Nvenc encoder for a while now and it's been pretty decent, but with the launch of the new Navi 2 cards, I'm interested in switching over but wanted to test the AMD encoder out first. With my Nvidia card, I've been using the following settings to record gameplay; Nvenc h.264, 1440p canvas and output, @60fps, CQP 15, and Maximum Quality. Usually for a 30-40min capture it dumps out a roughly 10GB file, recorded in MKV which I then run through Handbrake on almost placebo quality to get a MP4 which I then edit in Resolve 16 (free). I recently tried bypassing handbrake and just recording straight to MP4, but a 50min capture was 40GB. Not that I have an issue with the file size but that's quite a lot bigger.

Anyways... to test the AMD encoder, I tossed a 5700xt into my rig, and used the exact same settings, except obviously choosing the AMD h.265 encoder (as I hear AMD's h.264 is bad) and I also recorded in 1440p, 60fps, and CQP set to 15. The results I'm getting are not good... and I'm looking to find out if there is anything I can do. The primary issue I'm finding is that recording with Nvenc, Nvidia seems to retain much, MUCH more retail, and this is especially apparent in dark corner areas. With the AMD encoder, it seems to give extreme color banding? (if that's what it's called), in the darker areas. In outside enviroments, the detail seems almost the same, but since I've been playing the new Amnesia Rebirth game, I've been noticing the quality loss in dark area's when using AMD's h.265 encoder.

Also, fwiw, I did try recording using h.264 off the CPU, but even with faster preset in OBS, it's laggy as hell. Anything less, like fast, or medium will make things a power point presentation in the video playback.

So, here is a comparison shot between a dark area in Amnesia Rebirth, the leftside is Nvidia, and the rightside AMD. I know a picture doesn't do things justice but you can still see how Nvidia retains a lot more detail in dark area's. This isn't to say Nvidia's Nvenc is fault free, because I still notice color banding, but it seems to be far, far less, and the picture seems sharper with more detail. If you look at the floor you can see how Nvidia seems to retain detail where with the AMD recording it's really bad, and I'm hoping there are settings or things I can try to improve it to match Nvidia's nvenc.

View attachment 295505

And here's a 2nd shot of just AMD. You can see on the floor, close the barricaded furniture those dark areas look bad.

View attachment 295510
OBS support for AMD is rudimentary and they haven't even had a staff member for AMD stuff for a couple of months now. I would recommend trying another software to see if the H.265 quality is better. Such as AMD's own capture feature built into their driver. or Bandicam. Bandicam is payed, but its quite good for recording. AMD's H.265 is supposed to be solid.

There is no indication yet about the encoding quality of the new Navi cards

Davinci Resolve doesn't accept MKV as an input source?

If not, try MKV2Vob to switch MKV container to MP4 with no re-encoding. Its a great little program. AviDemux is also great. But if you just need swap the container, MKV2Vob would be faster.

Also, get StreamFX plugin for OBS Studio. It exposes Nvidia's H.265 for recording as well as even more quality options for H.264 and when set right, it rivals H.264 medium on CPU.
https://obsproject.com/forum/resources/streamfx-for-obs-studio.578/

download page
https://github.com/Xaymar/obs-StreamFX/releases/tag/0.9.1
 
Last edited:

Revenant_Knight

Limp Gawd
Joined
Nov 18, 2011
Messages
487
AMD is supposedly all hands on deck for driver and software now, and has shown substantial improvements over the last year. I suspect they will catch up. That said, recording with my 5700XT has been far more glitchy and troublesome then my RTX2080.
 

§kynet

Limp Gawd
Joined
Jun 11, 2003
Messages
414
Can you post a full sized image from both encoders so they can be directly compared?
 
Top