VR Eye tracking has really fired up everyone when they saw what opportunity it provides for virtual reality. Is it really as good as everyone is saying, or does it need some sacrifices in order to provide the benefits you hear about?
What benefits you ask?
Well, there are many, but let’s start with this:
Tell me what you see when you look 1 inch to the right from this WORD. My creative mind isn’t very creative tonight, so I’m writing nonsense with this paragraph. However, it does prove a point that peripheral vision doesn’t really work as well as one might imagine. You have to be looking exactly at the desired location to see it clearly. Although we do feel like things are super clear to us even in peripheral vision, it does blend everything together, making things hard to differentiate.
That is exactly one of the things eye tracking technology tries to accomplish – blur/lower everything in peripheral to increase performance, especially in VR. I’ll talk about this in a moment, but let me tell you in short how it got started.
It started with FOVE Headset
Gotta give credit where credit’s due. Back in 2015 FOVE created a lot of buzz surrounding eye tracking. Some even thought it would kill Oculus Rift seeing what advanced technology it packed. And at that particular moment it DID seem as though a dawn of new VR technology was upon us. The headset itself isn’t anything impressive, as it has slightly higher display resolution, and slightly lower FOV and refresh rate. But what did seem to market it to the top of the mountain was eye tracking technology.
Fast forward to 2017… what happened to FOVE? While it hasn’t disappeared and FOVE team is actively answering questions and releasing updates for this product, it has quieted down in terms of buzz. Reason? There are plenty, but the most probable ones are:
1. They suffer from a decease called “lackofcontent-itis”
It’s a terrible disease in VR that many have suffered, including Rift and Vive in their early days. They battled through it by having a lot of startup money for developers, and awarding developers for good content ideas. Although FOVE can access SteamVR (thank goodness to OpenVR), there aren’t a lot of games that support eye tracking, so that means, FOVE sadly remains a DEV Kit for the time being. Once eye-tracking technology gets traction, which will happen very soon, FOVE might come back guns blazing.
2. FOVE may have trouble competing against Rift and Vive.
Even if they fixed the content issue, fighting against such VR invested companies like Valve and Facebook might become an issue. Even the CTO of FOVE has said it’s a tough thing to compete against such giants. According to his words, FOVE better remains as a DEV Kit that can be used in places like showcasing or marketing a specific product that can be shown in VR. That’s why its price is 600 dollars even though they have no controllers.
What to do if FOVE is not good enough right now, and I want to play with eye tracking?
Implementation of eye tracking to current VR headsets. That is exactly what SMI, aGlass and Tobii – current leading eye tracking technology researchers – have done to Rift and Vive (mainly to Vive). For example, just around June 2017, Tobii has released eye-tracking technology development kit for HTC Vive. Tobii VR believes eye tracking is the next natural step for VR, and considering how many benefits it brings to VR users, I might say he is 100% correct. What are the benefits you might ask?
Benefits of Eye Tracking in VR
As far as my imagination and research go, there are 3 main benefits to gather from eye tracking.
Nr. 1 = Greatly improved interface in VR.
Eye tracking tracks your eye movement, and that means you can use that to your benefit in a number of ways. Want something to click? No problem, just look at it and blink (although blinking is not the best idea, as you blink naturally all the time). If you want to shoot something, just look at it and press the controller button. Thanks to the combination of accurate iris illumination, IR, raycasting, and through pre-processing to predict your eye movement, your eyes will be tracked super accurately.
Imagine what crazy ideas you can create through games and movies if eye tracking is implemented correctly? You can play around with depth of field, blurring, sounds…. options are only limited by your imagination. And that is only one of the benefits.
Nr. 2 = Social interaction.
If you have played video games at all, a stupid suggestion I know, you probably noticed that eyes are THE most unattractive and distracting qualities in AI models. It’s as if you are looking at a puppet that is doing whatever the script tells it to do. We are, of course, used to it and don’t bother ourselves with such details, but if we add eye movement to it, coupled with good hand tracking that Touch and Vive controllers provide, you truly notice a world of difference. Sometimes when the opposite gender looks at you, certain chemicals are released into the body, making you feel either butterflies in the stomach or just pure intensity. That is exactly what can be felt in VR now.
Check out this interesting video that was posted on r/vive:
Notice the difference? The difference is surprisingly huge considering how little was changed. That can be implemented anywhere from social games, or MMO VR games that require interacting with others. Like with interface, options are also limitless and provide great gaming immersion on top of virtual reality itself.
Nr. 3 = Optimization for games.
Now that is something I am most excited about. I have recently started to work on VR games (more to be announced soon by us), and having a good performance is a top priority. Every single VR game developer can tell you the same – if FPS is bad, the mood is bad.
As soon as VR eye tracking became a thing, developers started to research ways to use it to its maximum benefit. From that moment forward, foveated rendering was created. Foveated rendering is basically taking a huge load off the computer by reducing quality in places you are not looking at. The exact spot of your gaze is super clear, making it a fantastic option for developers who want to create scenes with more eyegasm.
The promise of this technology was so good that NVIDIA, biggest name in the gaming hardware industry, partnered with Sensomotoric Industries to advance it even further. In the beginning of the research, there were problems of seeing bad resolution on sides, even by looking with peripheral vision. After that they worked on making side picture more blurry, and that in turn managed to fake your eyes into believing it looks alright. However, even that wasn’t bulletproof due to ‘tunnel vision effect’ is created in the head.
That’s when “Contrast-Preserving Foveated Rendering” technique was created. As research has proven, our peripheral vision is mostly best at seeing the contrast, motions and edges, so what researchers at SMI and NVIDIA did was make side images high contrast, resulting in huge improvements in not only immersion but reducing the pixel shading workload by 3 times. That combination gives a significant performance boost in VR games, that lets developers make higher quality games.
And who wouldn’t like that?
But…
Are There Any Downsides to Eye Tracking in VR?
1) Well, if you are wearing glasses then you might run into some problems. Eye tracking technology for glasses, especially in Tobii’s side, has been improved upon, but you might still notice some mishaps from time to time. At this current moment, they are saying that it works on ‘most’ prescription glasses, but unfortunately not on all of them. A good alternative would be contact lenses as they don’t mess with eye trackers one bit, or keep an eye out for corrective lenses that some eye tracking companies provide.
2) One of the biggest problems in the VR industry is keeping the price down. Although VR hardware price is going slowly and steadily down, there is that issue of “having too many new VR inventions” for users. At the beginning you only needed a working Oculus Rift that cost 350 bucks, life was pretty simple. But then room scale tracking was added, additional 2 controllers to track, and now we talk about eye tracking or additional trackers for joints. It’s understandable if they were just accessories that “ADD” to the experience, but these days they are a “necessity”, without which most games won’t work.
Imagine if eye tracking became a necessary addition to VR package, then the price would go up even further, despite them trying to keep the price down. Luckily VR PC prices do go down, I even managed to make a VR PC build that costs only 600 bucks. I believe foveated rendering will lower it even further. However, these kinds of add-on truly make it hard for VR to enter the mainstream river (fast) when the cost of it all plays such a big role for people.
For example, aGlass upgrade kit for HTC Vive costs $220, and these are from China. Another company called Tobii is selling HTC Vive kits for developers where eye tracking is installed. Although the price for it remains uncertain, there are rumours of it adding additional $500 to Vive’s cost.
I hope as time goes on, they will make it more wallet-friendly, or have it already installed in VR headsets, just like FOVE has. I believe Rift and Vive are working on it, so I am truly excited for that.
3) The last problem might be a small one, but an issue nonetheless – they quality of eye tracking technology itself. There are different companies working on said technology, and I’ve heard different results and experiences coming from customers. Some have complained about latency, others are mentioning of peripheral vision being too blurry. NVIDIA partnering with SMI made a great progress in making it almost undifferentiable for users. But as I’ve seen from the latest news, not all companies have adopted it.
So keeping yourself aware of what brand is better will certainly improve everyone’s experience and thoughts about it.
The conclusion to VR Eye Tracking
I absolutely love it! As long as they figure out a way to keep the price down, I’m all for it. I am super excited about the performance benefits, as well as how you can use it artistically. I love when movies, animated clips, or even pictures are playing around with focus, and if eye tracking is well implemented, you can leave people in awe for years to come.
Social interaction is also something I’m interested to see flourish. As long as developers learn to master the VR creation every day, we’ll have something unbelievable in few years time.
Truly a great time to be alive!
If you want more VR related articles, feel free to share, follow us and support Rift Info!
