Skip to 0 minutes and 1 secondSo let's look at the first example exercise for artificial neural networks. Now, we've already looked at some neural network examples when we were going through theory and model analysis sections. So let's add something to the basic regression problem. Let's put some error bars or confidence intervals around the regression curve of the neural network model we generate. OK. So let's go to the code. As always, we start off by producing some data. This time, it's synthetic data. We're just estimating y based on x. We're going to split the data into training and test datasets.
Skip to 0 minutes and 49 secondsAll this is just normal housekeeping. Now we're going to create a neural network with three hidden nodes, remembering we need to get the formula. I want to estimate y based on x. Well, you'll find these variables in the training data. I want three hidden nodes. And it's a linear regression problem. It's not a linear regression problem, but it's a regression problem. So lin out equals true. Here's our model.
Skip to 1 minute and 19 secondsNow let's use that model to estimate the y values in our test data. Done. Now we can calculate the mean squared error of our model on the test data. There it is.
Skip to 1 minute and 43 secondsNow to get error bars or confidence intervals around our regression curve, we need the standard deviation of the residuals of the errors of the model on the test data. So let's get that.
Skip to 2 minutes and 0 secondsWe got that, incidentally. We just subtract the estimated y values from their true y values and then do the sd, standard deviation function. That gives the residual standard deviation. And now we're able to plot, first of all, the data. Then we will plot the regression curve of the model in blue. There it is.
Skip to 2 minutes and 37 secondsAnd now we'll plot the confidence intervals or regression curves plus or minus two standard deviations around the regression curve.
Skip to 2 minutes and 53 secondsAnd there we are. So we have a regression curve and confidence intervals around it for our model. This is a very nice an easy way of getting confidence intervals using the standard deviation. It's also possible, though, to do a more complex method, where instead of just doing standard deviations of the residuals, you actually plug the residual standard deviation into a normal distribution and specify the exact proportion probability you want between the two confidence intervals. So the standard deviations, typically, if you do one standard deviation, it's as if you're getting 0.68 probability. Two standard deviations, 0.95. Three standard deviations, approximately 0.995.
Skip to 3 minutes and 42 secondsBut we could specify other values as well by plugging this all into a normal distribution and actually calculating the confidence intervals that will give us a 90% probability of the true value being within those intervals. So let's calculate, for example, 90% confidence intervals in green.
Skip to 4 minutes and 5 secondsAnd you can go through and look how the two different methods are done, using standard deviations or using a normal distribution with the standard deviation as its parameter.
ANN Exercise 1
The first artificial neural networks exercise. The associated code is in the ANN Ex1.R file. Interested students are encouraged to replicate what we go through in the video themselves in R, but note that this is an optional activity intended for those who want practical experience in R and machine learning.
We create and evaluation a simple ANN model for a regression problem using synthetic data. We perform two-split hold-out validation. To add something, we also generate confidence intervals/error bars around the estimate of the MSE performance of the model on new data, using two different methods. Finally we plot the data, the regression curve and the calculated confidence intervals.
Note that the graphics, nnet and datasets R packages are used in this exercise. You will need to have them installed on your system. You can install packages using the install.packages function in R.
© Dr Michael Ashcroft