Want to keep learning?

This content is taken from the The University of Sheffield's online course, Making Sense of Data in the Media. Join the course to learn more.

Skip to 0 minutes and 9 secondsIf you're anything like me, you probably don't pay much attention to the terms and conditions that accompany many of our routine transactions. You simply agree to them without much thought and go about your business. But as we all know, there could be real consequences for failing to read the fine print. Imagine what would happen if you accepted a credit card offer without carefully considering the annual percentage rate, potential fees, and other details in the card's terms and conditions. As you might imagine, it'd be fairly easy to misuse your new credit card, racking up interest and fees, and potentially damaging your credit.

Skip to 0 minutes and 40 secondsMuch like this example, most published data sets will come with their own fine print, which is often described in the notes below figures and tables or in the methodology section for a longer document. This information is crucial to whether we should trust the data. Reputable organisations will provide some level of information about the data, either directly or easily accessible, so that we can see how it was collected, who procured it, and for what purpose. Without this basic information, we should be wary of any results drawn from the data. Before we look at a few examples of methodology reports in the media, let's review some important questions that you should ask any time you encounter a new data set.

Skip to 1 minute and 17 secondsThese questions are by no means exhaustive, but they do provide you with the key information you'll need to evaluate the merits of the data. Who sponsored the study? And who collected the data? These questions help identify whether there are any potential conflicts of interest and whether the work was carried out by a reputable organisation, known for high ethical standards and quality control. For example, we might want to scrutinise a survey if it was funded or collected by a partisan organisation. What is the sample size? Knowing how many observations are contained in the data set will help you decide whether you have enough evidence to properly answer your research question. When was data collected or observed?

Skip to 1 minute and 53 secondsIn other words, you need to know the relevant time period because we wouldn't want to draw conclusions from data that might be considered out of date or no longer applicable. Where is the data from? You need to ask whether the sample accurately represents the population to which we want to generalise. How was the data collected? The specific procedures that were used should be described in sufficient detail. For instance, if a survey is involved, how was it conducted? In person? By telephone? By mail or online? How about the response rate? Of those eligible respondents who were initially contacted, how many agreed to complete the survey? How well does the data capture the concepts?

Skip to 2 minutes and 31 secondsUltimately, this information helps you determine whether the study was conducted with rigour and the results should be trusted. Why was the data collected? Knowing the intended purpose of the data set should help you decide how it should be used or how it shouldn't be used. Some methodology descriptions in the media can be quite detailed. For example, The Washington Post provided a 400-word description of their methods for a survey about gun violence in the US. Other methodology descriptions are short, but still provide most of the key information. For instance, The Guardian presented the results from a poll about views toward the European Union and included a methodology section with just 43 words.

Skip to 3 minutes and 7 secondsAnd still other times, there is virtually no information about the data. For example, The Telegraph also presented results from a survey about the European Union and only provided the source of the data, nothing else. In this case, there simply isn't enough detail to evaluate the quality of the data. Ultimately, diving into the methods section can be quite daunting at first, but with a little practise, you can become a better data citizen.

Reviewing the methodology

Most reputable organisations will provide some level of information about the data they publish so that we can see how it was collected, who procured it, and for what purpose. Without this basic information, we should be wary of any results drawn from this data.

This information is often described in the ‘Methodology’ section of a document (i.e. news article, survey report, academic paper) and it is crucial to whether we should trust the data.

In this video, Todd explains some of the important questions a reliable methodology section should answer:

  • WHO sponsored the study, and who collected the data?
  • WHAT is the sample size?
  • WHEN was the data collected or observed?
  • WHERE is the data from?
  • HOW was the data collected?
  • WHY was the data collected?

Share this video:

This video is from the free online course:

Making Sense of Data in the Media

The University of Sheffield