NVIDIA RTX Video Super Resolution Tested, "Amazing" Video Upscaling in Your Browser | That Barely Makes a Difference

[erek]

Finally arrived:

"Similar to the well received NVIDIA Shield TV, which could take 720p or 1080p content and upscale it to 4K at up to 30 frames per second using the AI hardware within the Tegra X1+, RTX VSR is a further development of this technology. Using the more powerful hardware on modern RTX graphics cards, RTX VSR automatically upscales content played from within your browser between 360p and 1440p, to 4K, improving detail and removing the compression artifacts streamed content is known for.

NVIDIA's RTX VSR FAQ and blog post answers some common questions and provides further details on how the technology works.

You can take a look at NVIDIA's comparison video or try enabling the feature yourself to decide how well NVIDIA's efforts have paid off. As we've seen with other AI based deep learning solutions, the technology will continue to improve with time. In it's current state, RTX VSR seems particularly well suited for increasing the clarity of videos uploaded at lower resolutions or bitrates, such as older videos or live streamed content from Twitch or YouTube. Those using capped or slower network connections limiting their streaming options should also appreciate being able to efficiently consume content without sacrificing too much in image quality. I can't wait to see where the iterative path leads, as this technology could be as impactful in video media as AI based upscalers were for gaming!"



Source: https://www.techpowerup.com/305296/...ested-amazing-video-upscaling-in-your-browser

 

[LukeTbk]

simple Video example

 

[erek]

simple Video example

seems to be a giant "meh"

anyone seen anything mind blowing here?

i'm on native 4K IPS, and the comparisons video above does nothing magickal for me tbh

 

[schoolslave]

lol “upscaling” - no what we’re doing is taking guesses at how to fill in details left out by lossy (and/or too aggressive) compression. That’s not upscaling but instead making shit up - ie. upshitting.

Anyone else remember a fundamental of computing: GIGO?

 

[LukeTbk]

seems to be a giant "meh"

Not many people would pay anything to enable it from what I have seen, but at least the result is good enough to be quite obvious and not needing a label for you to know.

The difference between vsr2 and 4 too.... not so sure.

 

[LukeTbk]

lol “upscaling” - no what we’re doing is taking guesses at how to fill in details left out by lossy (and/or too aggressive) compression. That’s not upscaling but instead making shit up - ie. upshitting.

What you propose to happen if you have a 4k LCD type monitor and someone stream a less than 4k native signal ?

Without having an CRT, upscaling will necessarily happen, the decision is how-who does it (cpu-gpu, your monitor, etc...) not if you can't possibly not do it.

 

[schoolslave]

What you propose to happen if you have a 4k LCD type monitor and someone stream a less than 4k native signal ?

Without having an CRT, upscaling will necessarily happen, the decision of how-who does it (cpu-gpu, your monitor, etc...)

You can scale the content by duplicating pixels evenly in all directions or add vertical/horizontal bars and keep the original size. However, none of this addresses the fact that lossy compression by definition loses details so there is no way to get them back accurately without the original source - mathematically impossible. Period.

I’d rather watch the original content as-is than some made-up version based on AI models.

 

[Lakados]

What you propose to happen if you have a 4k LCD type monitor and someone stream a less than 4k native signal ?

Without having an CRT, upscaling will necessarily happen, the decision of how-who does it (cpu-gpu, your monitor, etc...)

There is also the case for legacy titles, what if you want to spin up some 20-year-old game on your new 32" 4K screen?

I have a large catalog that doesn't even support 16:9 let alone 4K, even playing those on a large 1440p screen looks bad.
This goes a long way to correct that issue.

 

[LukeTbk]

You can scale the content by duplicating pixels evenly in all directions or add vertical/horizontal bars and keep the original size. However, none of this addresses the fact that lossy compression by definition loses details so there is no way to get them back accurately without the original source - mathematically impossible. Period.

That seem obviously fall to me, to take a very easy example.

Say text with a known font was filmed, the video compressed and goes back to you, if something would be intelligent to recognize that it is text, the font and the letter it would be obviously possible to reconstruct the original, fully or in part, maybe ever better than the original video capture was able to capture it (thing what scanned document where able to do in the past, we could go from a word file, print it, scan it, read the scan and go back to better than the actual printed quality a perfect word file document), same for a stop sign or electric wire.
I think we will agree here ?
And I feel that would be true for human hair, eyes, building side, road side, cars, table and many other common trained pattern.

I’d rather watch the original content as-is than some made-up version based on AI models.

Well yes, (if both the sellers and you can afford it, it would be impractical) every video you consumme will be heavily compressed and decompressed by you to play back, the question being about the best way to do it, not if we will do it.

Say the vendor and me, agree on a 15mbs budget, it is quite possible, I think even likely that it can be better for it to send me 15mbs of 1080p and me to upscale it to 4k than for it to send me 4k directly instead, with that variable shifting to say around 50mbs where it would be better to send native directly (but would still be just has much compressed, has you send 4 time the pixels without 4 times more data)

 

[schoolslave]

That seem obviously fall to me, to take a very easy example.

Say text with a known font was filmed, the video compressed and goes back to you, if something would be intelligent to recognize that it is text, the font and the letter it would be obviously possible to reconstruct the original, fully or in part, maybe ever better than the video capture, same for a stop sign or electric wire.
I think we will agree here ?
And I feel that would be true for human hair, eyes, building side, road side, cars, table and many other common trained pattern.

Sure in this contrived case I agree with a caveat - we’re talking about visual reconstruction of a font or shape but we still could never exactly match the original source in terms of bits and bytes. Maybe that’s ok, especially in a case like this, but in general I think AI is a misapplication to upscaling/content “restoration” like this.

The proper application in my opinion is using AI to figure out the variables for highest compression to quality ratio per individual stream (based on heuristics like bandwidth budget, desired quality etc). There’s a reason Netflix and friends closely guard and protect their encoded streams compared to the decoded content. The compressed stream is worth “more” ie the “secret sauce”.

 

[OutOfPhase]

Sure in this contrived case I agree with a caveat - we’re talking about visual reconstruction of a font or shape but we still could never exactly match the original source in terms of bits and bytes. Maybe that’s ok, especially in a case like this, but in general I think AI is a misapplication to upscaling/content “restoration” like this.

The proper application in my opinion is using AI to figure out the variables for highest compression to quality ratio per individual stream (based on heuristics like bandwidth budget, desired quality etc). There’s a reason Netflix and friends closely guard and protect their encoded streams compared to the decoded content. The compressed stream is worth “more” ie the “secret sauce”.

Just having fun with this topic :

What streams do you have that have lossless streaming of original source material? Is the source material truly lossless?

 

[erek]

Just having fun with this topic :

What streams do you have that have lossless streaming of original source material? Is the source material truly lossless?

what about beyond apparent lossless toward Raw Uncompressed?

 

[GotNoRice]

Sounds like a great tech, but why did they have to ruin it by making it browser-only, and even worse, only available in chromium-based browsers?

 

[OutOfPhase]

what about beyond apparent lossless toward Raw Uncompressed?

Still going devil's advocate - is quantization loss not considered loss?

(Really not making a position here - just having fun with the idea of what is the source of truth)

 

[erek]

Not many people would pay anything to enable it from what I have seen, but at least the result is good enough to be quite obvious and not needing a label for you to know.

The difference between vsr2 and 4 too.... not so sure.

remember for the NV40 series when they charged for NVIDIA PureVideo? it was also catastrophically broken in silicon for some variants:

PureVideo Decoder https://www.nvidia.com/en-us/drivers/dvd-decoder/

NV40 broken video processor | [H]ard|Forum​

Oct 11, 2004 — The Video Processor (soon to receive a true marketing name) on the NV40 was somewhat broken, although it featured MPEG 2 decode acceleration

 

[schoolslave]

Just having fun with this topic :

What streams do you have that have lossless streaming of original source material? Is the source material truly lossless?

Good point - I consider whatever we had prior to being compressed and sent over the wire as the “source”.

 

[xenium]

Tested this with a 3080. Driving a 4k120hz monitor, every 1080p 60fps source I tested draws ~340w, at Quality 4 settings. In other words, be mindful when using this with high res/fps sources and high quality levels.

Sounds like a great tech, but why did they have to ruin it by making it browser-only, and even worse, only available in chromium-based browsers?

It works with MPC-HC and BE as well

https://github.com/emoose/VideoRenderer/releases/tag/rtx-1.0


Edit: For my own curiosity I tested all the settings/sources with a random youtube video. From what I've heard and personal experience, the difference between Quality 1/2 and 4 settings is nearly indistinguishable. YMMV, check it out yourself and decide if it's worth the hundreds of extra watts.

Note, the 1440p/Quality 4 result was hitting the ~375w power limit of the card. It would likely draw well over 400w if the PL was raised.

 

[GoodBoy]

How to Enable:
Install latest driver, if the option below is missing you need to update the driver.

 

[LittleBuddy]

I don't care much for this for streaming videos, because these days they are already high quality. But my question is, does this work with FMV's in games? Then I could see it being useful.

 

[madpistol]

it does make a difference, but not as much as I was hoping for.

 

[UnknownSouljer]

Just having fun with this topic :

What streams do you have that have lossless streaming of original source material? Is the source material truly lossless?

The answer is zero. Digital films used in theaters for distribution are 12-bit ProRes XQ files. They are on the order of multiple terabytes per film. No one streams that.

what about beyond apparent lossless toward Raw Uncompressed?

Extra zero.
First, RAW makes no sense as a distribution format as by its design it’s for doing work in post. It’s not a format that remotely looks like a final image.
Second, depending on the format and compression, as well as frame rate and frame size, RAW can easily exceed 1TB for 10 minutes of video.

One of Hollywood’s major costs at this point is just data management. A days worth of filming could be 10TB easily. And over the course of 90 or 120 days could exceed a petabyte.

It usually isn’t around this, simply because camera setup time is always what takes films so long to produce and not necessarily that they are running the camera constantly. A more conservative average is likely around 500TB, but that is still not a small amount of data to have to manage.

 

[LukeTbk]

One other way to quickly think about it.

HDMI 2.1 is 48 Gbps (sometime will send a compressed signal if you go high enough in fps-res, etc..), simple 1080p-60fps 8 bit color will be aorund 4.5 Gbps

Having 1 Gbps internet connection is considered ultra fast most place
A 100GB 4kUD bluray will be 0.15Gbps including the rather big audio
high quality stream will be around 0.025-0.035 Gbps

The idea of something outside production step arriving to your screen uncompressed does not make much sense (or the notion that Xmbs of bandwith+ AI upscaling cannot beat the same bandwith at native, really does not) we are always talking how, by who, how much, etc...

 

[erek]

The answer is zero. Digital films used in theaters for distribution are 12-bit ProRes XQ files. They are on the order of multiple terabytes per film. No one streams that.

Extra zero.
First, RAW makes no sense as a distribution format as by its design it’s for doing work in post. It’s not a format that remotely looks like a final image.
Second, depending on the format and compression, as well as frame rate and frame size, RAW can easily exceed 1TB for 10 minutes of video.

One of Hollywood’s major costs at this point is just data management. A days worth of filming could be 10TB easily. And over the course of 90 or 120 days could exceed a petabyte.

It usually isn’t around this, simply because camera setup time is always what takes films so long to produce and not necessarily that they are running the camera constantly. A more conservative average is likely around 500TB, but that is still not a small amount of data to have to manage.

isn't even RAW Uncompressed digital still a sampling (rate and bit-depth) of reality anyhow?

you're only capturing so much information

---

"Bhusal et al. developed a quantum camera that exceeds the Abbe-Rayleigh resolution limit for super-resolution imaging.

Utilizing the self-learning capabilities of artificial intelligence, their approach for quantum statistical imaging identifies the statistical fluctuations of totally unknown mixes of light sources." / "Recently, there has been interest in using spatial projective measurements to enhance the resolution of imaging systems. Unfortunately, these schemes require a priori information regarding the coherence properties of “unknown” light beams and impose stringent alignment conditions. Here, we introduce a smart quantum camera for superresolving imaging that exploits the self-learning features of artificial intelligence to identify the statistical fluctuations of unknown mixtures of light sources at each pixel. This is achieved through a universal quantum model that enables the design of artificial neural networks for the identification of photon fluctuations. Our protocol overcomes limitations of existing superresolution schemes based on spatial mode projections, and consequently provides alternative methods for microscopy, remote sensing, and astronomy."

Source: https://www.nature.com/articles/s41534-022-00593-5

 

[gerardfraser]

I am seeing GPU power 2x-3x with Nvidia Super Resolution for minor difference in off vs on imgsli screenshots

♦ CPU - Intel 13700KF
♦ GPU -GIGABYTE GeForce RTX 4090 GAMING OC

Microsoft Edge
Version 110.0.1587.57 (Official build) (64-bit)

Windows 11, version 22H2
(OS Build 22621.1265)

Super Resolution set to highest #4 in 3 screenshot

2560x1440 screen Video source youtube crap on purpose 1920x1080 video
https://imgsli.com/MTU5MjQ5

3840x2160 screen Video source youtube crap on purpose 1920x1080 videos
https://imgsli.com/MTU5MjU0
https://imgsli.com/MTU5MjYw

 

[LukeTbk]

I am seeing GPU power 2x-3x with Nvidia Super Resolution for minor difference

I think most of the minor difference can be achieved with the VSR-2 without much power usage, the last image seem to be the only one that has a big visible effect,w as it running at the max vsr-4 ?

 

[M76]

lol “upscaling” - no what we’re doing is taking guesses at how to fill in details left out by lossy (and/or too aggressive) compression. That’s not upscaling but instead making shit up - ie. upshitting.

Anyone else remember a fundamental of computing: GIGO?

That theory was true until a few years ago. Since AI all bets are off. It can actually guess the missing content quite well. We used to laugh at zoom and enhance cliché, now it is actual reality.

 

[gerardfraser]

I think most of the minor difference can be achieved with the VSR-2 without much power usage, the last image seem to be the only one that has a big visible effect,w as it running at the max vsr-4 ?

Super Resolution set to highest #4 in 3 screenshot

 

[TheSlySyl]

It's not that noticeable on most talking head 1080p videos, but you get some really shitty tiktok or mobile uploads that have been compressed to hell and you can see an immediate difference.
However it's not really a "better" difference, it looks overly smooth and *weird*, as they're obviously trying to hide the blockiness of the poor compression.

Also it doesn't want to work on my chrome, but it does work on edge. I assume its due to some of my chrome youtube plugins?

 

[erek]

I think most of the minor difference can be achieved with the VSR-2 without much power usage, the last image seem to be the only one that has a big visible effect,w as it running at the max vsr-4 ?

Intel Outs Video Super Resolution for Chromium Browsers, Works with iGPUs 11th Gen Onward​

Intel Graphics has introduced feature rivaling NVIDIA VSR (virtual super resolution) for Chromium-based web-browsers such as Google Chrome and Microsoft Edge, which enhances the quality of streaming video, both from lower resolutions to higher; as well as improving the quality at native resolution. This was discovered in the change-log of a Chrome build, which talks about the introduction of a "-features=IntelVpSuperResolution" command-line argument to start Chrome with, to enable the feature.

 

[erek]

How to Enable:
Install latest driver, if the option below is missing you need to update the driver.
View attachment 553017

"This video compares the new NVIDIA RTX Video Super Resolution (VSR) feature, testing 720p, 1080p, 1440p, and 480p to see if the feature can reconstruct a higher quality image with a lower data requirement. The testing includes comparisons of GN videos against source 4K material as well as Twitch streamer content (like Asmongold/zackrawrr, Pieman, EVGA, etc.) of different types. This includes FPS games and slower-moving games like Elden Ring, which have different responses to NVIDIA RTX Video Super Resolution technology."

 

[MaZa]

You can scale the content by duplicating pixels evenly in all directions or add vertical/horizontal bars and keep the original size. However, none of this addresses the fact that lossy compression by definition loses details so there is no way to get them back accurately without the original source - mathematically impossible. Period.

I’d rather watch the original content as-is than some made-up version based on AI models.

If you have a choice to watch original in high quality then of course that is always preferable, nobody questions that. But if that is not an option and all we have is, for example, DVD grade or even lower video? Then I would rather watch it through an intelligent upscaler that atleast tries to sharpen the image while also detects edges smooths them into straight or curved lines instead of simply stretching the crappy image even bigger. Fine details are forever lost but edges do not have to look like staircases or whole screen coated in vaseline smear like old dumb binaural and bicubic scalers do.

 

[TheSlySyl]

I eventually turned it off. I watch a lot of YouTube on a second my monitor while I game and the GPU usage was too damn high, was killing my framerate in Minecraft (RTX, and no not a joke.) And also messing with my plex Transcoding.

That, and, honestly, it just made things look worse. The smoothing effect it had was was more distracting than the default low-res look.

 

[MaZa]

Excellent news for HTPC users like me. Super Resolution is no longer limited for browser use as latest version of MPC Video Renderer (by Aleksoid) added support for it. MadVR used to be the king of upscalers but IMHO Nvidia Super Resolution exceeds it.

480p without proper scaler vs Nvidia VSR comparison. The difference is stunning.
https://imgsli.com/MTU4MzM1

480p upscaled with MadVR's NGU Sharp vs Nvidia VSR. Very close but Nvidia takes the cake in my eyes.
https://imgsli.com/MTU4NTg4/0/2

Link for the latest version of MPC Video Renderer, used with MPC-BE or newer forks of MPC-HC video players
https://github.com/emoose/VideoRenderer/releases/tag/rtx-1.1

 

[MaZa]

I have to add though. On high quality 1080p content VSR is not actually preferable. There the slightly vaxy feel makes the image worse and you lose film grain that should be there on the video. Stick to MadVR for those. But on low quality 1080p or 720p with compression artifacts it works really well on removing the said artifacts without blurring the image. And on DVD quality video the results are just stunning. Below DVD though, well, there is only so much upscaling can do...