Principles and Applications of Three Reality Representation Techniques in Virtual Reality Technology

The Visualization of Reality in VR An important theme in the concept of virtual reality is the exploration of how the richness of the real world can be transformed into a digital space and make it possible for us to interact with it. Computer scientists and engineers have developed a plethora of different technologies over the past few decades that allow us to immerse ourselves in the infinite details and diverse landscapes of the real world through virtual display windows. These technologies are becoming more and more common in our lives, whether in education, healthcare, entertainment or engineering, and can expand the boundaries of reality through virtual reality.

This session will take you into the heart of these amazing technologies. From capturing real scenes to reconstructing detailed virtual spaces, from simulating tactile interactions to generating spatial sound effects, we will gradually unravel the mysteries behind these technologies. Through easy-to-follow explanations, we hope you can understand in just a few minutes how virtual reality allows us to experience another kind of reality without leaving home.

The first thing we’re going to cover is one of the simplest VR technologies: panoramic picture technology. Panoramic picture VR technology is a method of generating an immersive virtual reality experience using panoramic photographs, through which the user can look around from a fixed point and experience a full perspective of the scene. The core of this technology lies in the 360-degree photography of the scene, enabling the user to have an immersive viewing experience in a VR device. The technique is simple to obtain and immersive even with very low hardware performance. However, because images are static, they do not allow for coherent spatial interaction. Developers often try to add hotspots to pictures to present richer media content.

If you want to reproduce a real scene in virtual space with a higher degree of detail, Oblique Photography modelling is a good choice! Oblique Photography modelling technology is a technique for 3D modelling of terrain and buildings by taking aerial photographs from multiple angles. All you need to do is to take as many pictures as possible from different angles and input them into a professional 3D reconstruction software to generate an accurate 3D model of the scene. Oblique Photography modelling technology not only accurately reproduces the geometry of the scene, but also preserves the real texture at the same time. Oblique Photography modelling technology has been used in a number of games to significantly enhance the realism and immersion of the graphics. For example, in the Black Myth ：Wukong developed by Game Science, developers have used the technology to restore a large number of real buildings and objects to the virtual game space.

With the development of AI technology, more convenient workflows have emerged in recent years, enabling us to quickly bring realistic scenarios into virtual reality environments, and one of the most iconic technologies is Neural Radiation Field technology. NeR is a neural network-based rendering technique that allows complex 3D scenes to be generated from a small number of 2D images. In contrast to Oblique Photography modelling technology, NeRF requires only a few photographs to render highly photorealistic images and is able to reproduce the material characteristics of objects more accurately.

In fact, there are already very accessible options for porting your surroundings to the game engine for virtual reality content development using NeRF. We will use luma AI and Unreal Engine as an example to show you how to bring the actual scene around you into Unreal Engine.

You’ll first need to download the latest version of Unreal Engine, then head over to Fab, Unreal Engine’s resource mall, and download the luma AI plugin, a tool that will help us render the Neural Radiation Field screen in Unreal Engine. After that, visit Luma AI’s webpage, log in to your account, upload a video of your environment that you have taken under the guidance of the system, and wait for some time for Luma AI to convert the video you have taken into a file in the specific format of the neural radiation field. By opening Unreal Engine and importing your Luma Field file, you can drag and drop that resource directly into the world, and you’ll be able to freely observe the NeRF scene in the viewport of the unreal engine at this moment. More notably, NeRF scenes can receive lighting information from the Unreal engine for adaptive rendering effects, which will greatly aid our virtual reality content development. Despite NeRF’s high fidelity and adaptability, there are still some problems.NeRF requires high computing capacity and is mainly applicable to static scenes, for dynamic objects NeRF performs poorly.

The characteristics of each of these three technologies make them suitable for different VR application scenarios, ranging from the rapid display of panoramic picture VR, to the high-precision reconstruction of tilt photography, to the highly realistic presentation of NeRF. With the advancement and integration of technology, they will bring richer content and experience to the field of virtual reality. The purpose of this course is to help you understand the basic principles and application scenarios of these technologies, and in the future, we hope that you can choose the right VR technology according to your specific needs, and provide users with a more realistic and immersive virtual experience.