Nicholas Jewell on quantifying uncertainty

Well, I think the huge advantage of surveys, as you say it, scientifically and statistically is by conducting them using well-understood techniques of sampling, that is, in fact, what allows you to directly [INAUDIBLE] to a measure of uncertainty, assuming everything else went fine. So you get the uncertainty from the fact that I only sampled 1% of the population. And you understand how you sampled that 1%. That allows you, when you extrapolate up to describing an estimate for the entire population, also to extrapolate a measure of uncertainty based on just the sampling properties.

You’re absolutely correct, though, that that in itself is a narrow version of uncertainty in that it doesn’t account for things that went a little bit wrong or choices that were made in the midst of taking the survey. How you deal with non-response would be a classic example of that. So the non-respondents don’t– the uncertainty estimates don’t maybe go up or down depending on the level of the response, but how much the estimate may be biassed in terms of how non-respondents might have responded is not accounted for by sampling uncertainty. And of course, yet that fundamentally is part of what we don’t know.

And so statisticians have gotten more and more sophisticated in trying to deal with things like non-response or missing data with fairly weak assumptions to say there’s not only uncertainty because we’re only sampling a piece of the population, but there’s also uncertainty because the group we sample don’t all respond. And then there’s uncertainty because the group that do respond don’t tell the truth, and so on and so forth. And these errors, of course, promulgate. And part of my job, and part of statisticians’ job, is trying to, as honestly as we can, account for all of these sorts of error, not just a sampling variation.