Skip main navigation

Comparing classifiers

Ian Witten discusses how to reliably compare two classifiers. This involves rejecting the "null hypothesis" at a given statistical significance level.

How can you reliably compare two classifiers? Experimental results always depend on the random number sequence – might the conclusion be different if you used a different random number seed? Statisticians talk about the “null hypothesis”, which is that one classifier’s performance is the same as the other’s. We’re usually hoping that the results of an experiment reject the null hypothesis! This involves a certain level of statistical significance: we might reject the hypothesis at the 5% level of statistical significance, meaning that it’s highly unlikely (1 chance in 20) that their performance is the same.

This article is from the free online

More Data Mining with Weka

Created by
FutureLearn - Learning For Life

Reach your personal and professional goals

Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.

Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.

Start Learning now