£199.99 £139.99 for one year of Unlimited learning. Offer ends on 14 November 2022 at 23:59 (UTC). T&Cs apply

Find out more
What happens when different errors have different costs?
Skip main navigation

What happens when different errors have different costs?

Ian Witten introduces this week's second Big Question

In this course (and its predecessor, Data Mining with Weka), we’ve been obsessed with classification accuracy – the percentage of correct predictions on test data – as the measure of success.

But is it really the right thing to measure?

We touched on this when looking at document classification in Week 2, where we learned how to use “threshold curves” to show different tradeoffs between error types. But it’s not just document classification. In reality, when you look at the larger picture surrounding any deployment of machine learning, different errors almost always have different costs. Even with the weather: if today I undertake a certain activity because suitable weather is predicted, and that prediction is incorrect, I might suffer, even die. This – though thankfully not the dying part – happened to me just the other day, when I was sailing offshore in predicted 20 knot winds and was hit by an unforecasted 50 knot gust. (It was exciting, and dangerous.) On the other hand, if unsuitable weather had been incorrectly predicted, I might just have spent the day playing music instead of sailing, in which case little would have been lost.

At the end of this week you will be able to take different error costs into account when doing machine learning. Of course, we won’t be able to help you ascertain what the costs are. But if you know the different costs of incorrect predictions, you’ll be able to take this into account when measuring performance. You’ll also know how the output of an ordinary classifier can be post-processed to take account of different error costs, and – an alternative method – how to build a classifier that takes account of the costs internally.

This article is from the free online

More Data Mining with Weka

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education