Testing the validity of your beliefs

It’s extremely difficult to test the validity of your assumptions and data gathering instruments – especially when we’re talking about qualitative data.

But it’s not impossible.

Here are a few approaches you can take.

Question the validity of your focus groups, surveys and research subjects

Constantly ask yourself, at every stage of the process, if your research could have any inconsistencies. When you find them, try to find ways to iron them out.

If you only conducted focus groups at 2 pm on weekdays, you did not get a representative sample.

If you only paid participants £25 for a two-hour session, there’s no way your focus group is representative.

Design your questions to test for truthfulness

In focus groups and surveys, people are aware that they are being watched and judged.

You can’t make them tell the truth, but there are plenty of methods to test for inconsistencies.

Questionnaire design is a deep, complicated topic that is outside the scope of this course. However, this article from the Pew Research centre is a great place to start.

If you conduct a lot of qualitative research or intend to start, hire a professional survey designer or researcher.

Have survey questions and focus groups validated by relevant peers and experts Just like any written work, our surveys and focus groups can end up littered with loaded questions and inherent biases if we’re not careful.

Get as many people as you can to review your surveys before you use them. The more diverse the group of people, the better – yet another reason to hire more diverse people.

Ensure that at least some of the people validating the test are actually from the target group

There are severe limitations in having a middle-aged, white male from London validate a qualitative questionnaire for young people from Uttar Pradesh.

Conduct multiple studies with similar groups and compare results.

If results vary wildly between groups, try to figure out why. Your initial hypothesis might be totally off the mark. If so, at this point, you’ll have enough research to come up with a new one.

Book several focus groups at different times and places.

People who can come to a focus group at 2 pm on a Tuesday will likely have different opinions than people who can only come at 6 pm on a Friday.

Iterate your approach to interviews based on what you learn.

Ask questions like:

• What times did we hold these focus groups?
• How much notice did we give?
• How much did we pay participants?

If your target groups don’t respond well to your questions, redesign the survey based on the answers previous groups have already given.

Do I really need to do this?

Some of these approaches may seem like overkill – especially if you work in a fast-paced startup with limited resources and even less time.

But if you want to make representative work, you need representative data. If you want to get representative data, you need representative samples.

Most importantly, if you want to have a representative interpretation of that data, you must have a diverse team to interpret it.