Skip main navigation
Home / IT & Computer Science / AI & Robotics / Applications of AI Technology / Virtual-to-Real Learning

Virtual-to-Real Learning

v
View transcript
14.4
After the success of DeepMind, researchers start to explore other applications using deep reinforcement learning. Remember that DeepMind uses Atari games to demonstrate deep reinforcement learning. After that, researchers have started to use more powerful 3D engines, and applied virtual learning techniques to other computer vision applications, including image segmentation, indoor navigation, autonomous driving, and many others.
44.8
There are two examples below: The UnrealCV used Unreal engine to render labelled object data for learning semantic segmentation; CAD2Real trained models in a virtual world created by AutoCAD, and successfully deployed on a real drone. Most important of all, researchers found that the models trained in virtual world can be deployed in real world. Let’s see some examples from Pan’s BMVC paper. The pictures on the left side are virtual images, and the pictures on the right side are real images, As we can see, the virtual images and real images are almost the same after semantic segmentation.
88.3
That shows how easily we can deploy models trained in virtual world to real world Since the virtual-to-real learning has a lot of potentials, big companies and researchers start to build virtual environments for training deep learning models. Microsoft creates Project Malmo and AirSim Project Malmo is based on Minecraft. which is a very flexible game that allows users to build their own world and create their own game rules. Facebook created House3D, HoME based on Matterport’s 3D scanning and reconstruction technology. DeepMind built DeepMind Lab based on Quake, an old 3D shooting engine (game) Intel created MINOS, which is also based on Matterport technology Unity created Unity AI using their own 3D engine and created tasks which are like OpenAI Gym’s.
145.2
There are also many open-source environment from Academia. The most popular one is OpenAI gym. We also support OpenAI gym API for deep reinforcement learning. As we can see, there is a need for open-source real-world simulator. To fill this gap,
163.4
we developed an open-source VR engine called VIVID: Virtual Environment for Visual Deep Learning. Our work won the best open source software award of ACM Multimedia 2018. VIVID is based on the Unreal engine.

What is the real world look like for drones? We can see from the slide. Prof. Lai will talk about applications using deep reinforcement learning.

Nowadays, big companies and researchers start developing many virtual environments Industry:

  • Microsoft: Project Malmo (Minecraft), AirSim
  • Facebook: House3D, HoME
  • DeepMind: DeepMind Lab
  • Intel: MINOS
  • Unity: Unity AI

In Academia also there are a couple famous example.

  • OpenAI Gym
  • Deep GTA V (DeepDrive)
  • AI2-THOR, Sim4CV, UnrealCV, Stanford 3D, VizDoom

Want to keep
learning?

This content is taken from
Taipei Medical University online course,

Applications of AI Technology

View Course

From the example above, there seems a need for companies or researchers to develop or improve virtual environments more. Thus, an open-source real-world simulator becomes important. Prof. Lai will continue to introduce VIVID: Virtual Environment for Visual Deep Learning.

Want to keep learning?

This content is taken from Taipei Medical University online course
Applications of AI Technology
View Course
See other articles from this course
This article is from the online course:

Applications of AI Technology

Created by
Join Now
This article is from the free online

Applications of AI Technology

Created by
Join Now
FutureLearn - Learning For Life

Reach your personal and professional goals

Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.

Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.

Start Learning now

Register to receive updates

  • Create an account to receive our newsletter, course recommendations and promotions.

    Register for free