Skip main navigation

How do we design good experiments?

How do we design good experiments?
© Coventry University. CC BY-NC 4.0

While we may have access to lots of existing data, it does not necessarily mean that it will answer the questions we want to ask.

Sometimes we will need to go out and collect more data. This can be expensive and time-consuming. This is why it’s important to make sure you have the right experimental design in place before you get started; to make sure you collect the right data and enough of it to answer your questions and avoid some of the possible problems outlined in previous steps.

There are some key stages in the experiment design process:

  1. Have one or more clear and explicit hypotheses generated from a good understanding of the problem. This will help enormously in scoping your design and in turning stakeholders’ questions into testable experiments.

  2. Think about what you are comparing. This is often to assess the effect of a proposed decision or improvement, so, for example, the baseline or current situation may be compared with the improvement. Additionally, consider what else might affect the comparison. Could, for example, the time and place the data is collected or the people in two groups you are comparing differ in important ways? You will need to factor this in.

  3. Think about the best way to collect the data you need. This may be available from enterprise analytics systems or you may need to conduct a survey or design a customised interface to collect it.

  4. Ensure that your sampling – the data you do collect – is representative of the groups you want to say something about.

  5. Think about how much data you will need to be in a position to differentiate any effect from randomness/chance. For example, if a coin flip gives us heads five times in a row, does it mean the coin is biased or is this just normal randomness?

A note on randomness. Randomness is really interesting. In practice, it often behaves in unexpected ways and probably explains a lot of findings that we mistake to be due to our own action or skill. As humans, we have evolved with senses that are tuned to recognising patterns quickly (eg seeing a poisonous snake in the grass), but this means that we also can be fooled into seeing patterns that are not there (ie it’s just a random patch of grass that looks a bit like a snake).

Further reading

Spiegalhalter, D. (2019). The art of statistics. Pelican.

© Coventry University. CC BY-NC 4.0
This article is from the free online

Get ready for a Masters in Data Science and AI

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education