Skip main navigation

End-to-end autonomous driving

What rules do self-driving cars need in order to naviagate in highly complex real-world road environments? Watch Dr Will Smith explain more.
We’ve seen some examples of using simple rules to navigate in very simple environments. But what about in the highly complex real-world road environments that a self-driving car must navigate? Here, we must deal with other moving vehicles, pedestrians, bikes, multiple lanes, road signs, traffic lights and so on. Performance must be reliable whether it’s day or night, sunny or raining, foggy or snowing. Trying to deal with such complexity using hand engineered rules quickly becomes impossible. We’ve already seen how deep convolutional neural networks can be used to predict a value or class from an input image.
Because we learn the mapping from image to desired output using real data, our system learns robustness to the sorts of clutter, noise and complexity that we encounter in the real-world. We’ll now see how this approach can be powerful enough to provide basic self-driving capability in real world environments. The idea is actually very simple. We feed into our network frames from a video camera that provides a drivers-eye view of the road. The network will learn to predict the correct steering controls for the current situation. This might be as simple as what angle the steering wheel should be rotated and how much the brake or accelerator should be applied.
Then, to drive, we simply connect the neural network to the camera and driving control interface and allow it to map the video camera feed into steering controls. We call this end-to-end autonomous driving since the neural network is learning the entire process, from one end (input images) to the other end (driving controls) with no hand engineered systems in between. The first and most obvious question
is: where do we get the training data to train such a system? Actually, this is, in principle, straightforward. We equip a car with a camera and some sensors to measure the state of the steering controls. Then, we have humans drive the car and store images along with the driving controls that the human applied when the image was taken. These form training data pairs for our system. Surprisingly, such a simple approach actually works, at least to a limited extent. A car trained like this can stay in its lane, take corners, slow down at intersections and cope with a fairly wide range of environmental conditions. However, it cannot provide reliable driving over longer time periods.
A single camera may not provide enough information about the environment, processing each frame independently can lead to an erratic driving style and there are many limitations of a fully end-to-end approach. Finally, the system has no goal - we have not provided a destination and the system cannot form a route plan using a map.

What rules do self-driving cars need in order to naviagate in highly complex real-world road environments?

Watch Dr Will Smith explain more.

This article is from the free online

Intelligent Systems: An Introduction to Deep Learning and Autonomous Systems

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education