Machine learning vs deep learning

What are the key differences between deep learning and the machine learning techniques we have seen in previous weeks?

This article gives a brief summary.

Machine learning

The figure below summarises the key steps we might take in all of the machine learning methods we have seen in weeks one to four of the course.

Pre-processing

We start with some raw data, such as a set of digital images or a table of numerical data, after which there is a pre-processing step. If the dataset is complete and well organised the pre-processing might be as simple as loading it into memory and storing it in an array. In other cases, there could be missing data that needs to be dealt with, or with image data they might need to be reshaped to be of uniform size. In general, this step consists of any action you need to take to ensure your dataset is both complete and in a uniform format suitable for further analysis.

Feature extraction

In previous weeks, we’ve talked quite a bit about feature extraction. This is any step you take with your raw data where you combine or transform your original data features to make new features that will make better (or additional) features as input to a machine learning model. For example, this could be image edge extraction, such as HOG feature extraction, or some sort of dimensionality reduction such as PCA.

In general, good feature extraction requires knowledge in the subject area of your data, expertise in the machine learning algorithm you are intending to use, and in many cases a lot of trial and error.

Machine learning algorithms

Once you have extracted your features the next step is to select a machine learning algorithm and use it to train a model to your data. We’ve seen quite a few in the course, but there are many more we haven’t had time to introduce. Which algorithm you select depends on both your data and on the type of problem you’re looking to address.

For regression this could be linear regression or decision tree regression, for clustering this could be K-means, for classification this could be Naive Bayes, and so on. You can even use a neural network model in this same way, as we’ll see in the practical later this week. Once the model is trained you can use it to make predictions on new data and evaluate how well it is working.

Deep learning

We can summarise the steps in deep learning as in the following figure.

We still start with our raw data and may need to do some pre-processing before training a model. There are two big differences here, however. Firstly, there is no feature extraction step. A particular strength of deep learning techniques is that it can operate directly on raw data and effectively learn the features that need to be extracted during the training process itself.

The second big difference is that there isn’t really a choice of which algorithm to use. In deep learning you will generally be using some sort of neural network model, in particular a kind of neural network known as a convolutional neural network (CNN). We’ll talk a bit more about CNNs later in this week of the course.

The trade-off with this apparent simplicity in the overall process is the complexity of the model itself. There’s a big variety of CNNs in use, and all have their strengths and weaknesses, so selecting the type and size of CNN requires care. In addition, this increase in complexity leads to challenges both in terms of producing the code needed to run them, and in the time taken to train them. We’ll talk more about the advantages and disadvantages of deep learning approaches once we have looked in more detail at how they work.