What methods can be used in online experiments?

As you do online experiments with people and conduct usability testing at scale, it is important to ensure that you always have a solid research question. Too often, organisations collect data without a clear sense of why they are doing it, or what it means.

Just collecting that data is not enough; focusing on identifying insights that will help your users or organisation to work more efficiently and effectively will focus your time and attention.

A few other key points to keep in mind when carrying out online experiments are:

A/B testing and preference testing are not the same

A/B testing requires that you have a baseline understand of what is ‘normal’ for your experience and that you want to test alternative approaches that will drive an improvement in defined metrics. This means that it is exclusively conducted on live sites.

Preference testing can be conducted at any stage in the development process, once you have a concept, wireframe, or other representation of the experience to show to study participants.

Sometimes, organisations will conduct both preference and A/B testing. If there are too many options to test using your optimisation solution, you might opt to do preference testing to rule in or out options to test in on the live site.

Qualitative and quantitative scale

Remote, unmoderated testing can be done at both qualitative and quantitative scale. When you want to focus on validating usability metrics such as task completion, success, and satisfaction, time on task, or error rate, testing with a larger number of participants will help you to measure, compare, and make the case that a design approach is working.

If your intent is to iteratively test and improve an approach, using smaller numbers of participants – typically between five and eight – will help you to gather the insights you need to make improvements.

Usability testing basics

Usability testing basics apply in a remote setting too. When setting up a script, think about the order of tasks, getting informed consent, and remaining neutral as a moderator while prompting participants effectively.

Guiding people effectively through a test without giving them too much direction is a delicate balance.

Using search analytics

Search analytics are an often unexplored goldmine of interesting and useful information. Looking at what terms people are using to access and navigate your site gives you a great deal of insight into what they are thinking is there. You can get ideas about what content to add – or remove because people are not looking for or using it.

In addition, ideas of labels, messaging, or ways to position products or other content may come out of this line of inquiry. Look at not only the top terms people use, but also be sure to explore that terms out in the “long tail” – those that are used less frequently.

Often, though the terms themselves might be used less frequently, there is more variety here, and you can start to find relationships, themes, or trends.