Skip to 0 minutes and 10 secondsWhat is a survey? And why do we conduct them? To avoid controversy, I'll answer this question within the context of an innocuous example, abortion. Suppose we want to know the percentage of the US population that supports abortion. Our first idea is simply to ask people what they think. This makes sense, right? But now, two obvious questions jump out at us. One, who do we ask? And two, how do we ask them? One answer to the first question is that we could ask every single person in the whole country. But there's a glaring problem with this proposal. It would be terribly expensive to do this. It's much cheaper to just interview some people.

Skip to 1 minute and 3 secondsBut then we need to be careful about how we pick the people to interview. Interviewing people as they walk out of an abortion clinic will almost surely lead to different results from what we would get if we interview people as they walk out of a church. The group of people we actually interview is called our sample. My point about churches and abortion clinics is just my way to say that we want our sample to be representative of the whole population that we're trying to understand. Ideally, our sample will be a microcosm of the US. That is, abortion support within the sample should be equally strong as abortion support within the country as a whole.

Skip to 1 minute and 47 secondsThe question then becomes, how do we get such a representative sample? We'll return to this issue shortly in the course. Let's now return to question number two. How do you elicit people's views about abortion? In fact, the reason I picked the abortion example in the first place is that it's easy for anyone to think up loaded questions that would push the responses in one direction or another. So I won't even bother here to propose some loaded questions. Instead, I'll show you a subtle case where a seemingly innocuous change in an American survey question seems to have a fairly big effect on people's answers. Question 1, abortion should be legal in almost all cases. Agree or disagree? 28% ticked agree.

Skip to 2 minutes and 40 secondsQuestion 2, women should have a legal right to safe and accessible abortion in almost all cases. Agree or disagree? Now, 37% agree. Conclusion, you need to know the exact wordings of questions that are asked in surveys before you can begin to interpret a survey. Always ask to see the questions.

The Idea of a Survey

After I recorded this video a veritable volcano blew its stack on social media in a way that is highly relevant to my last point about knowing exact wordings of questions.

A YouGov poll asked people about their prospects for winning a tennis point against Serena Williams.

The responses to this question can and has been discussed vigorously (to put it mildly) from many angles. You’re all free to express your own views in our discussion forum as long as you follow our rule to be nice to each other (which was not observed very much on social media).

Here I make a relatively narrow point that extends my video discussion. Different people can interpret a single question in different ways so you should be careful about assuming that you know what question various survey respondents are actually answering.

Here is the YouGov question:

“Do you think if you were playing your very best tennis, you could win a point off Serena Williams?”

12% of men and 3% of women answered “yes”. There’s a rather big gender difference in the responses that’s worth discussing but I won’t do so here. Instead, I draw your attention to YouGov’s interpretation of the finding that they tweeted out, thereby launching the whole firestorm:

“One in eight men (12%) say they could win a point in a game of tennis against 23 time grand slam winner Serena Williams”

I see two main problems with this interpretation.

  1. First, if “Serena Williams” had been preceded by “23 time grand slam winner” in actual question that YouGov asked then I’m pretty sure that fewer people would have answered “yes”.

  2. Second, and more importantly, the actual question does not place a limit on the number of chances you get to win a point. Yet YouGov’s interpretation places a very low limit, specifically 4 chances, on your wiggle room. That’s a huge difference. Surely, some people thought something like the following: “if Serena serves 1,000 times eventually she’ll double fault so my best answer is ‘yes’”.

The author of the linked article gives third interpretation.

“…the research firm ask[ed] British citizens whether they believed that they would be capable of winning a point in a tennis match against Serena Williams.”

A match yields considerably fewer chances than 1,000 but far more chances than 4. In any case, the question says nothing about a match.

Another wrinkle is that in online discussions some people emphasized the phrase “playing your very best tennis” which is in the actual question wording. In my opinion, this bit is only relevant to a tiny handful of people who are extremely good tennis players. But I don’t really know how other people might interpret and rate the importance of this phrase.

Discussion

Speculate on what was actually measured by this question and propose further research that could test your speculations. And, please, be nice to each other!

Share this video:

This video is from the free online course:

Survival Statistics: Secrets for Demystifying Numbers

Royal Holloway, University of London