Skip main navigation

Scene perception in autonomous driving

Is camera data enough to navigate the highly complex real-world road environments safely? Dr Kofi Appiah discusses what other technology is needed.
We’ve seen how video data can be used in autonomous driving. If we assume the camera data to be the eagle eyes of the self-driving car, would that be enough to navigate the highly complex real-world road environments safely?
Think of yourself as a driver: do you make use of other senses like hearing whilst driving? Maybe we need more than one video camera, and maybe we need different types of sensors besides video cameras for a safe self-driving car. Sensors are key components of self-driving cars. You can argue that having more sensors will make autonomous vehicles safer. However, having many sensors comes with the challenge of fusing all the data together correctly and accurately, bearing in mind that the data may not have been collected at exactly the same time or at the same frequency.
A typical prototype autonomous vehicle may have multiple cameras - LIDAR, RADAR and SONAR, which all measure distance using different types of wave - a GPS system for location and accelerometers and Inertial Measurement Units for measuring motion. While a camera might be very useful for identifying and tracking objects, information about the exact distance of the object from the vehicle will be best estimated using range sensors like LiDAR or RADAR. For self-driving cars to be deployed in real-world driving environments, they must be capable of safely interacting with nearby pedestrians and vehicles, and thus having the ability to detect and track moving objects.
The camera can be used to identify the class of the moving objects (such as car, pedestrian, cyclist), but to have an accurate sense of how fast they are moving and the direction of movement, a LiDAR or RADAR sensor is helpful. For example, RADAR can provide position and velocity information of an object approximately 200 meters away. Compared to video cameras and LiDAR, RADAR is least affected by extreme weather conditions like snow and fog, and above all, requires minimal data bandwidth and processing power. In contrast to RADAR, which uses radio waves, LiDAR uses light waves and provides 3-dimensional data points in space, dense enough to delineate the shape of objects.
LiDAR can map static environments as well as detect and identify moving vehicles, pedestrians, and even wildlife. By using a rotating sensor, it is able to model the environment across the full 360 degree view. Combining data streams from LiDAR and RADAR provides more reliable information about the distance of objects in the environment. To fully understand the scene, motion data from RADAR and LiDAR can be used to segment regions of interest such as potential obstacles or moving objects. These can be passed to a central unit where all the sensor data is fused and used in conjunction with the camera data for object classification.
Once objects are labelled and their positions accurately known, higher level driving control systems can decide how to react accordingly.

Is camera data enough to navigate the highly complex real-world road environments safely?

Dr Kofi Appiah discusses what other technology is needed.

This article is from the free online

Intelligent Systems: An Introduction to Deep Learning and Autonomous Systems

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education