Skip main navigation

Testing the validity of your beliefs

Research processes and focus groups are rife with researchers' assumptions. Here are a few things you can do to mitigate their effects.
Beach pebbles fill the screen, but when you look a little closer, in the bottom right, a single pebble has a question mark on it.

It’s extremely difficult to test the validity of your assumptions and data gathering instruments – especially when we’re talking about qualitative data.

But it’s not impossible.

Here are a few approaches you can take.

Question the validity of your focus groups, surveys and research subjects

Constantly ask yourself, at every stage of the process, if your research could have any inconsistencies. When you find them, try to find ways to iron them out.

If you only conducted focus groups at 2 pm on weekdays, you did not get a representative sample.

If you only paid participants £25 for a two-hour session, there’s no way your focus group is representative.

Design your questions to test for truthfulness

In focus groups and surveys, people are aware that they are being watched and judged.

You can’t make them tell the truth, but there are plenty of methods to test for inconsistencies.

Questionnaire design is a deep, complicated topic that is outside the scope of this course. However, this article from the Pew Research centre is a great place to start.

If you conduct a lot of qualitative research or intend to start, hire a professional survey designer or researcher.

Have survey questions and focus groups validated by relevant peers and experts Just like any written work, our surveys and focus groups can end up littered with loaded questions and inherent biases if we’re not careful.

Get as many people as you can to review your surveys before you use them. The more diverse the group of people, the better – yet another reason to hire more diverse people.

Ensure that at least some of the people validating the test are actually from the target group

There are severe limitations in having a middle-aged, white male from London validate a qualitative questionnaire for young people from Uttar Pradesh.

Conduct multiple studies with similar groups and compare results.

If results vary wildly between groups, try to figure out why. Your initial hypothesis might be totally off the mark. If so, at this point, you’ll have enough research to come up with a new one.

Book several focus groups at different times and places.

People who can come to a focus group at 2 pm on a Tuesday will likely have different opinions than people who can only come at 6 pm on a Friday.

Iterate your approach to interviews based on what you learn.

Ask questions like:

  • What times did we hold these focus groups?
  • How much notice did we give?
  • How much did we pay participants?

If your target groups don’t respond well to your questions, redesign the survey based on the answers previous groups have already given.

Do I really need to do this?

Some of these approaches may seem like overkill – especially if you work in a fast-paced startup with limited resources and even less time.

But if you want to make representative work, you need representative data. If you want to get representative data, you need representative samples.

Most importantly, if you want to have a representative interpretation of that data, you must have a diverse team to interpret it.

This article is from the free online

Cultivating Diversity through Research and Insights

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education