2.15

## The University of Waikato

Skip to 0 minutes and 11 secondsHi! Good to see you again. One of the things I like to do with my time is play music, and that little bit of Mozart you hear at the beginning of these videos, that’s me and three friends playing a clarinet quartet I play in an orchestra, and last night I was playing some jazz with a little trio. If you want to hear us play, if you go to Google and find my home page, type my name, “Ian Witten”, you’ll get me here, and every time you visit this page

Skip to 0 minutes and 53 secondsIf you refresh the page, I’ll play you another tune.

Skip to 1 minute and 5 secondsThat’s what I do. Anyway, that’s not what we’re here for. We learned about cross-validation in the last lesson. I said that cross-validation was a better way of evaluating your machine learning algorithm, evaluating your classifier, than repeated holdout, repeating the holdout method. Cross-validation does things 10 times. You can use holdout to do things 10 times, but cross-validation is a better way of doing things. Let’s just do a little experiment. I’m going to start up Weka and open the “diabetes” dataset. Here it is, diabetes.arff, and the baseline accuracy, which ZeroR gives me – that’s the default classifier, by the way, rules/ZeroR – if I just run that, well, it will evaluate it using cross-validation.

Skip to 1 minute and 59 secondsActually, for a true baseline, I should just use the training set. That’ll just look at the chances of getting a correct result if we simply guess the most likely class, in this case 65.1%. That’s the baseline accuracy. That’s the first thing you should do with any dataset. Then we’re going to look at J48, which is down here under “trees”. There it is. I’m going to evaluate it with 10-fold cross-validation.

Skip to 2 minutes and 32 secondsIt takes just a second to do that. I get a result of 73.8%, and we can change the random-number seed like we did before. The default is 1; let’s put a random-number seed of 2. Run it again. I get 75%. Do it again. Change it to, say, 3 (I can choose anything I want, of course), run it again, and I get 75.5%. These are the numbers I get on this slide with 10 different random-number seeds. Those are the same numbers on this slide in the right-hand column, the 10 values I got, 73.8%, 75.0%, 75.5%, and so on.

Skip to 3 minutes and 18 secondsI can calculate the mean, which for that right-hand column is 74.5%, and the sample standard deviation, which is 0.9%, using just the same formulas that we used before. Before we used these formulas for the holdout method, we repeated the holdout 10 times. These are the results you get on this dataset if you repeat holdout using 90% for training and 10% for testing – which is, of course, what we’re doing with 10-fold cross-validation. I would get those results there, and if I average those I get a mean of 74.8%, which is satisfactorily close to 74.5%, but I get a larger standard deviation, quite a lot larger standard deviation of 4.6%, as opposed to 0.9% with cross-validation.