How does Virtual Reality work?

In the previous step, examples of both Virtual Reality (VR) and Augmented Reality (AR) were introduced. Before we go on to discover more about immersive experiences, we need to understand what VR/AR actually is, and how this technology works.

Virtual Reality

“The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.” – Ivan Sutherland (1965)

This quote from 1965 envisioned a future with the ultimate display, where the environment and on display is so realistic, that it simulates a user’s physical presence within that environment. This ultimate display, potentially has a lot of features of what we now describe as Virtual Reality.

Virtual Reality is a computer interface where graphics look real, audio sounds real, interactions feel real, (and perhaps even) smells and tastes real.

There are lots of different application areas for VR, including:

• Simulation: experiences that allow people to experience expensive or dangerous activities.
• Journalism: experiences that help viewers to empathise by getting a sense of ‘being there’.
• Product design: allowing users to inspect and manipulate designs as if they are real, and of course;
• Games: enhancing the realism crucial to the experience of many game genres.

But, how does VR actually work? At present, VR systems work by tricking our senses into thinking we are in a different environment:

1. Vision

Displaying 3D images of a virtual world that encompass the user’s whole field of view.

2. Touch

Force feedback devices that recreate sensations of touching objects.

3. Hearing

Headphones provide 3D representation of sound from the virtual world.

At the moment, this is what current VR systems are limited to. But, it would also theoretically be possible to expand this to include our sense of:

4. Smell

Olfactory output devices that recreate smells from the virtual world.

5. Taste

Taste output devices (which are yet to be invented!) that recreate the taste of objects in the virtual world.

Most VR experiences are implemented by combining the following key components:

• Head-mounted 3D display – to let the user see,
• Spatial audio: to let the user hear,
• Locomotion technique: to let the user move
• Interaction technique: to let the user manipulate.

To learn more about the first three of these key components, check out the linked articles.

When it comes to interaction techniques, in an ideal world interactions (e.g. with objects) in the virtual world would be indistinguishable from interactions in the real world. This means that there would be realistic physical expression where interactions with virtual objects mimics the physical objectives with similar complexity and precision. And, importantly, these interactions would need realistic responses and provide realistic feedback to the user, e.g. by creating realistic sounds, and so that the user can feel the objects as well as see and hear them. As it stands, realistic interactions still need some work! The technology available now does allow us to create perfectly indistinguishable interactions. However, while existing techniques don’t allow for perfect replications of physical interactions, we can still make compelling interactions that give the user a sense of immersion while working within the constraints of modern technology. Users are willing to suspend their disbelief with useable interactions that aren’t completely realistic, where:

• Less realistic + more useable = more immersive
• More realistic + less usable = less immersive

And this is why, although people will often use handheld controllers within VR experiences, they are still able to become completely immersed in the experience, in spite of the lack of perfect reality of the interactions.

In VR, we can theoretically see anything, go anywhere and do anything. However, VR doesn’t truly simulate reality yet. There are issues with VR experiences that should not be ignored. For example, they can be:

• Isolating: it’s hard to interact with other people wearing a Head-Mounted Display.
• Touch-less: haptic technology is quite far away from recreating a realistic sense of touch.
• Limited movement: You can’t walk and run around. Locomotion techniques are usable, but do they feel real?
• Asset heavy: virtual worlds need to be created, and the bigger and more complex these virtual worlds are, the longer it takes.
• Immobile: much VR technology requires infrastructure, so you can’t use it out and about .
• Disconnected from reality: VR puts us in an alternative reality, but what if the user wants to use digital technology in the context of their ‘actual’ reality?

This leads us to think about Milgram’s Reality-Virtuality Continuum.

• In an Augmented Reality experience, the real world is augmented with aspects of the virtual world.
• In Augmented Virtuality, the virtual world is augmented with aspects of the real world.
• In Virtual Reality there is an opportunity to create experiences that cannot exist in the real world using digital technology.

Over to you

Have you ever used Virtual Reality? What did you think of the experience? What type of locomotion technique did you use? Did you find it gave you any motion sickness? Or any other negatives?