Facebook Reality Labs Unveils Its DeepFocus AI Rendering System

Facebook Reality Labs (FRL) has announced DeepFocus; an AI rendering system that works in conjunction with its upcoming eye tracking hardware called Half Dome. Half Dome has a varifocal design that that will recognize when users focus on objects in the VR world. DeepFocus AI will recognize this action and perform a render blur on the surrounding areas to simulate the way a human eye works.

The "rendered blur" effect that the DeepFocus AI performs is essential to creating a convincing VR image. The more realistic the rendered blur effect is, the more natural the scenes in VR appear to the user. When our eyes focus on an object, other objects at different depths appear blurry. The researchers at FRL are attempting to recreate the same effect with DeepFocus. The goal of the project is to create a more comfortable VR experience that can be used all day and "work for every single title in the Oculus Store, without asking developers to recompile."

"This is about all-day immersion," says Douglas Lanman, FRL's Director of Display Systems Research. "Whether you're playing a video game for hours or looking at a boring spreadsheet, eye strain, visual fatigue and just having a beautiful image you're willing to spend your day with, all of that matters."

I don't think people want the stuff they're not directly focused on to be blurry. I thought the point was to save processing power by only having a small portion of the display be in hi res.

meme said:

I don't think people want the stuff they're not directly focused on to be blurry. I thought the point was to save processing power by only having a small portion of the display be in hi res.

Click to expand...

That's the approach that SteamVR is taking I believe.

meme said:

I don't think people want the stuff they're not directly focused on to be blurry. I thought the point was to save processing power by only having a small portion of the display be in hi res.

Click to expand...

They are addressing two separate problems
Problem 1: Rendering high resolution stuff for VR is hard. Can we render only what the person is directly looking at in high detail? Who cares about the rest of the stuff. (this is not related to depth)
Problem 2: Fully immersive stereo 3d requires 2 things. The most important, showing two separate viewpoints and having your eyes cross at varying angles to allign the images is already implemented and works pretty well. The 2nd thing is When you focus on something at a certain depth, everything at other depths becomes defocused (blurred). When you are showing a 2d image to each eye this effect is not recreated. So this research is on how to accurately and quickly recreate that effect.

Problem 2 can actually be broken up into two more separate subproblems:
a) Objects in the virtual world are all at the same focus no matter which one your looking at it and no matter what depth they are at. This is what the linked research is addressing.
b) You are used to exerting force to change your eye lens shape to actually focus on different objects. When you are staring at the same screen 2 inches from your face your eye does not need to do this anymore, and it may feel a little strange when you look at something up close and then at somewhere far away in the virtual world and your eyes don't have to refocus. They talk about this in the beginning of the siggraph facebook video. About having a system where a variable lens changes shape based on what you are looking at.

As long as what Serpretetsky has explained is the case here, more power to them.

All I know is that the way games currently handle DoF, is shit. "Not what's being focused on? Aaallright. Engage Blur Shader!" *sigh*
That's just not how eyes work, and while I can appreciate that it'd require a lot more processing power to do it right, I'd rather it just not be done at all. To do it right is basically what Serp had explained, as it deals with the fact we have two eyeballs doing work to achieve depth and focus. When I am looking at my screen on my laptop, there's plenty in the background, and while it is 'blurry', it's not exactly in the same way that games render the blur. Furthermore, and key in my opinion, is that the background (or foreground, depending on the case) imagery is doubles... due to our pair of eyes

Fallout 76 has some heavily enforced DoF post processing and I seriously hate it. Again, as Serp mentioned, our brain is wired to adjust when something is our of focus and we want to focus on it... yet the game has its own thoughts on the matter and even with my crosshairs pointed at <whatever>, it's not fully in focus. So here I am, SQUINTING UNCONTROLLABLY at my damn TV (monitor) trying to focus on something I can't, purely because that's how eyes work and I can't help it. Then I catch myself doing it and force myself to stop, only to once again be immersed in the gameplay and the viscious cycle repeats! lol Ugh

cageymaru Video down for anyone else? Removed by user on my end.

Formula.350 said:

As long as what Serpretetsky has explained is the case here, more power to them.

All I know is that the way games currently handle DoF, is shit. "Not what's being focused on? Aaallright. Engage Blur Shader!" *sigh*
That's just not how eyes work, and while I can appreciate that it'd require a lot more processing power to do it right, I'd rather it just not be done at all. To do it right is basically what Serp had explained, as it deals with the fact we have two eyeballs doing work to achieve depth and focus. When I am looking at my screen on my laptop, there's plenty in the background, and while it is 'blurry', it's not exactly in the same way that games render the blur. Furthermore, and key in my opinion, is that the background (or foreground, depending on the case) imagery is doubles... due to our pair of eyes

Fallout 76 has some heavily enforced DoF post processing and I seriously hate it. Again, as Serp mentioned, our brain is wired to adjust when something is our of focus and we want to focus on it... yet the game has its own thoughts on the matter and even with my crosshairs pointed at <whatever>, it's not fully in focus. So here I am, SQUINTING UNCONTROLLABLY at my damn TV (monitor) trying to focus on something I can't, purely because that's how eyes work and I can't help it. Then I catch myself doing it and force myself to stop, only to once again be immersed in the gameplay and the viscious cycle repeats! lol Ugh

Click to expand...

Without seeing the vid, I'd imagine its similar to what (I think, Intel?) does with pro-CS at times with the eye tracking stuff, you can see the players checking that corner off their currently held angle, quick glances at the map at times, etc., and the software adjusts the games DoF around that?

I'd be inclined to agree with your second paragraph, does the game really need to assist me with what my eyes do naturally on their own? I'd doubt it can keep up (without some SERIOUS horsepower) on my eyes darting around the "screen", changing focus to match my will?

Idk, interesting, sure. But I'd prefer to let my eyes take care of the DoF without the need for AI assistance. Can't imagine that's comfortable for long periods, and unnerving when it doesn't work as intended.

AlphaQup said:

cageymaru

I'd be inclined to agree with your second paragraph, does the game really need to assist me with what my eyes do naturally on their own? I'd doubt it can keep up (without some SERIOUS horsepower) on my eyes darting around the "screen", changing focus to match my will?

Idk, interesting, sure. But I'd prefer to let my eyes take care of the DoF without the need for AI assistance. Can't imagine that's comfortable for long periods, and unnerving when it doesn't work as intended.

Click to expand...

When your eyes are staring at a screen they do not naturally focus on to "far" and "close" virtual objects. They simply adjust focus to the flat screen, which is at a single depth regardless of what you are looking at in the virtual world. This means when you are wearing a VR headset your stereo vision is in constant conflict with your eye focus compared to what you are used to in the real world. It may or may not be noticeable. The few times I tried it I was aware that my focus was going screwy. But maybe for some people it's not a problem. Shrug.

AlphaQup said:

cageymaru Video down for anyone else? Removed by user on my end.

Click to expand...

I fixed it. For some reason they changed the video in the article. You can view it now.

Excellent, thank you!

serpretetsky said:

When your eyes are staring at a screen they do not naturally focus on to "far" and "close" virtual objects. They simply adjust focus to the flat screen, which is at a single depth regardless of what you are looking at in the virtual world. This means when you are wearing a VR headset your stereo vision is in constant conflict with your eye focus compared to what you are used to in the real world. It may or may not be noticeable. The few times I tried it I was aware that my focus was going screwy. But maybe for some people it's not a problem. Shrug.

Click to expand...

Never really thought of it that way, that certainly makes sense though. Idk if I'd like something taking care of my DoF though, I already have enough problems with that in VR at times.

Idk, slightly off topic (maybe), but I've always had issues with DoF/"double vision" at times in VR. I do have a much stronger left eye then right (muscle is just much weaker, been that way all my life), and am right handed (makes firing pistols/shotguns/AR's a bit more difficult, whole head has to come across to get my left eye involved, but hey, still a good shot ) but yeah, VR always does throw me off at times having the screen so close. Not sure I want something more f'ing with that while I'm already struggling on occassion. I do get fatigued easily too when in it.

Anyone else find DoF ugly and distracting and has purposefully turned it off in every game they've ever played with it on?

WBurchnall said:

Anyone else find DoF ugly and distracting and has purposefully turned it off in every game they've ever played with it on?

Click to expand...

Yep lol That's why I'm annoyed at how ingrained it is with Fallout 76 They added a new slider in the options to adjust it but it didn't seem to do anything that I could tell (I do have a couple custom INI settings/tweaks that may be the "blocking" it somehow). When sitting still and just looking at the scenery, sure... DoF looks nice... Because you're just sitting there! Any sort of gameplay and it becomes a headache, both metaphorically and literally.

However, curiously, I've started to like using Motion Blur, where before I hated it. Or maybe just Dying Light did it so well and that's why I enjoyed it *shrug* heh