How does Augmented Reality work?

In the last step, we looked at Virtual Reality experiences and got a taster for how VR technology works. At the end, we discussed the Reality-Virtuality Continuum and defined the concepts of Augmented Reality (AR) and Augmented Virtuality. In this article, we’re going to look at AR in more depth.

So, we know that AR integrates digital information into a real-world environment. AR allows users to:

• Move freely, without fake locomotion
• Experience expressive interaction with real, physical objects
• Socialise and collaborate with others in real space
• Make strong connections between digital content and objects that exist within the physical world

From a technological perspective, it means that developers only need to create assets that augment the physical world, and not the entire environment

To create a convinced AR experience, we need a method to display digital content to the user so it appears to augment physical space, and a method to register that content so it corresponds with features of physical space.

Some examples of AR experiences include:

• Pokémon Go
• IKEA AR Catalogue
• FXMirror
• Hololens Project X

For these AR experiences to work – where virtual content takes physical space into account – we need an accurate 3D representation of the environment, and knowledge of the user’s position and orientation within that environment. There are numerous different ways to approach this from a technological perspective, including: device pose, Fiducial markers, body tracking and SLAM (Simultaneous Location and Mapping.).

Pokémon Go uses the device pose approach, where the virtual content is kept in the same coarse location by tracking the pose (i.e. orientation and rotation) of the device.

The IKEA AR Catalogue uses Fiducial markers where known images are placed within a camera frame, and by looking at the shape of the marker in the camera image we can calculate its position and orientation.

The FXMirror uses body tracking where the user’s limbs are tracked using a depth camera, and content is them overlaid onto a skeleton.

Finally, the Hololens Project X uses SLAM – Simultaneous Location and Mapping. With SLAM, a depth camera captures a 3D scan of the area of the environment that the user is currently looking at. A 3D model of that environment is then constructed, and the user’s location and orientation within the environment is computed.

All of these approaches work to augment the physical world with virtual content, and each approach has various advantages and disadvantages. But, ultimately, the end result is a mixed reality experience that combines the physical and virtual worlds.

Now that we know how VR and AR work, in the next step we’ll delve deeper into finding out how museums and galleries are using this technology to allow us to engage with, and better understand, the past.

Over to you

Have you ever used Augmented Reality? What did you think of the experience? Do you have any ideas of how these new technologies of AR and VR could be used to create exciting, engaging, immersive storytelling experiences?

As always, share your thoughts in the comments below!