Skip to 0 minutes and 13 seconds Well, I think the huge advantage of surveys, as you say it, scientifically and statistically is by conducting them using well-understood techniques of sampling, that is, in fact, what allows you to directly [INAUDIBLE] to a measure of uncertainty, assuming everything else went fine. So you get the uncertainty from the fact that I only sampled 1% of the population. And you understand how you sampled that 1%. That allows you, when you extrapolate up to describing an estimate for the entire population, also to extrapolate a measure of uncertainty based on just the sampling properties.
Skip to 0 minutes and 58 seconds You’re absolutely correct, though, that that in itself is a narrow version of uncertainty in that it doesn’t account for things that went a little bit wrong or choices that were made in the midst of taking the survey. How you deal with non-response would be a classic example of that. So the non-respondents don’t– the uncertainty estimates don’t maybe go up or down depending on the level of the response, but how much the estimate may be biassed in terms of how non-respondents might have responded is not accounted for by sampling uncertainty. And of course, yet that fundamentally is part of what we don’t know.
Skip to 1 minute and 44 seconds And so statisticians have gotten more and more sophisticated in trying to deal with things like non-response or missing data with fairly weak assumptions to say there’s not only uncertainty because we’re only sampling a piece of the population, but there’s also uncertainty because the group we sample don’t all respond. And then there’s uncertainty because the group that do respond don’t tell the truth, and so on and so forth. And these errors, of course, promulgate. And part of my job, and part of statisticians’ job, is trying to, as honestly as we can, account for all of these sorts of error, not just a sampling variation.
Nicholas Jewell on quantifying uncertainty
We have just expended a fair amount of energy learning about the quantification of uncertainty surrounding sample survey-based estimates of violent war deaths. So perhaps some of us are not currently in the best frame of mind to receive Nicholas Jewell’s message that there is still more uncertainty that our methods do not account for. But receive it we must.
You may recall that I made the same point back in step 2.4, saying that I mentally adjust all error margins by doubling their width. This is an arbitrary rule of thumb but it does suggest the importance that I attach to Nicholas’ main point in this clip.
© Royal Holloway, University of London