Skip main navigation

New offer! Get 30% off one whole year of Unlimited learning. Subscribe for just £249.99 £174.99. New subscribers only T&Cs apply

Find out more
Home / Business & Management / Data Science / Survival Statistics: Secrets for Demystifying Numbers / Potential Pitfalls

Potential Pitfalls

In this video Professor Michael Spagat presents some of the main things that can go wrong in surveys. But what can go wrong doesn't have to go wrong.
View transcript
10.4
Nobody’s perfect. And surveys pull together the work of lots of somebodies. So we should not ask whether a particular survey is flawed. Instead, we should ask how it’s flawed and how we should adjust our interpretations to accommodate these flaws. It’s no more possible for me to list all the things that can go wrong for a survey than it would be for me to list all the positive qualities of my cat Ernie. There’s always going to be another one. Still, here’s a list of four that are particularly important, with the last two applying just to election and referendum prediction. One, respondents may not give correct answers to the questions they are asked.
55.8
For example, in UK-based surveys men report having substantially more opposite gender sex partners on average than females do. But this is impossible. Think about it. Similarly, people often tell little white lies to surveyors about their voting, church attendance, and exercise. Two, a sample may be systematically unrepresentative of the broad population we’re trying to understand. For example, let’s say we want to estimate average income in the UK, but we do this by sampling from the members list of the British Lawn Tennis Association. Our estimate will probably be too high. And our systematic bias towards selecting people with above average incomes would remain no matter how large we make our sample.
105.1
Because we’ll always be selecting from a list of paid up tennis club members. Now let’s turn to the last two items on my list of potential pitfalls, which recall, are specific to predictions of election or referendum results. Three, it’s not good enough to understand people’s preferences for candidates. You also need to predict who is going to turn up and actually vote. Consider the UK’s Brexit referendum of 2016. Older people were much more likely to favour leaving the EU than younger people were. So it was important to understand likely turnout rates broken down by age. We know that in general older people are more likely to vote than younger people are. But how strong is this effect?
155.8
Our answer is pretty important for our predictions. Or consider the Trump versus Clinton election of 2016. It appears that white voters without college degrees, a very pro-Trump constituency, were more likely to turn up and vote, then many pollsters had assumed when making their predictions, which are largely based on turnout patterns in recent elections. The surprisingly high engagement of this pro-Trump constituency may have been enough to tip the balance in a few key swing states. Number four, people may change their minds after responding to a poll but before an election. Again Trump-Clinton provides a good example. You may recall that just before the election, FBI director James Comey announced that he was reopening the FBI investigation into Clinton’s emails.
216.2
This announcement seems to have hurt Clinton, even though shortly thereafter Comey announced that the new investigation didn’t find anything. One final and particularly important point– margins of error do not capture any of the potential pitfalls I’ve discussed in this clip. And the clip itself is just a highlight reel of some of the biggest errors that can mess up surveys and election predictions. Traditional margins of error are really just minimal underestimates of the true uncertainty that looms over survey estimates. Full uncertainty can easily be twice as large. So be careful. Think through the problems that can distort each survey finding you encounter, and treat error margins as the start of a conversation not the finish.

Think back to steps 2.12 and 2.13 about error margins. Notice that none of the pitfalls covered in the video are even mentioned in these steps on error margins. Error margins and the pitfalls reflect two separate sets of considerations.

Steps 2.12 and 2.13 are about uncertainty caused by the fact that we interview only a sample of the population rather than the entire population. This sampling issue, at least as presented in these steps, doesn’t systematically “bias” our estimates upwards or downwards compared to the true numbers we are trying to estimate. It does, however, inject uncertainty into our estimates.

In contrast, the issues raised in the present video do systematically “bias” results in one direction or another. For example, a sample packed with rich people will tend to lean Tory or more Republican more than the population as a whole does.

Discussion

Want to keep
learning?

This content is taken from
Royal Holloway, University of London online course,

Survival Statistics: Secrets for Demystifying Numbers

View Course

What can you add to my list of things that can go wrong with surveys and bias the results in a direction you can identify?

Caution: I’m not suggesting that everything that can go wrong will go wrong. Think of it like this. When you’re planning a family vacation it might be a good idea to first sit down and list all the things that might go wrong. The list might look a little daunting but this doesn’t mean that you should just stay home. In fact, making such a list should increase your chances of making a successful trip.

Want to keep learning?

This content is taken from Royal Holloway, University of London online course
Survival Statistics: Secrets for Demystifying Numbers
View Course
See other articles from this course
This article is from the online course:

Survival Statistics: Secrets for Demystifying Numbers

Created by
Join Now
This article is from the free online

Survival Statistics: Secrets for Demystifying Numbers

Created by
Join Now
FutureLearn - Learning For Life

Reach your personal and professional goals

Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.

Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.

Start Learning now

Register to receive updates

  • Create an account to receive our newsletter, course recommendations and promotions.

    Register for free