Analyzing Moderated Tests

When you have collected all of your data, the best part of research starts – figuring out what it all means. This brief reading will provide some tips and tricks for how to determine when a pattern is really a pattern or whether a finding is significant enough to include in a report.

Making meaning out of qualitative data can be more challenging than situations where you have structured data such as surveys to work with; the work that one would do ahead of time to anticipate the best way to phrase and frame questions as well as the answer choices participants would expect to see require much thought and planning upfront. When you are working to elicit and bring meaning to unstructured data, that work happens after those interactions – whether in person or remote – occur. Sometimes, you cannot be sure what you will learn until you start reviewing your data.

As you review the videos of the tests you conducted, or the transcripts of the interviews you conducted after the tests, use the following strategies to analyze the data and generate findings.

1. Get familiar with your data – review your recordings at least twice. Whether this means watching videos or reading transcripts, it is important to expose yourself to the entire experience participants had during the test to be sure you know what happened. This is also a great time to start taking quick notes; you might use the Notes option in the UserTesting video player, paper, or even post-its to record your impressions, ideas, and observations. You might discover that some of your data is not good; participants may not have been thoughtful, provided good feedback, or your script lead them in directions you did not anticipate. In a professional setting, you would run more tests to gather good data, sometimes after having tweaked the test itself to correct for any deficiencies. For this assignment, if this is the case, write your report about what you expected to find based on your research questions and why you think your sessions did not provide the results or findings you expected. As you do, consider the following questions:

Did participants do what you expected? Why or why not? Did participants say one thing and do something that did not reflect what they said? This might include rating something well when they had difficulty, clicking on elements that you do not expect, etc. Why? What behaviors are surprising? Why? What information do participants have or not that helped them to complete the tasks? Did everyone know that information? Is it really a problem, an inconvenience, or an inconsistency? What does it mean? What can you do about it? 2. Make sure your research questions are still valid. While you crafted and ran the study, you might have evolved your original research question or drifted away from what it was – often for good reasons. Review what you found and determine whether or not you can still answer these questions. If not, think about what you can learn given the data you have collected and report out on what you have learned. In a professional setting, you would need to evaluate whether your methodology was sound, whether your research question was answerable, or determine how to adjust your approach.

1. Focus your analysis. Create a framework for yourself that you can use to identify themes, patterns, or trends. Options include: Events of interest (clicks on elements, missed clicks, etc.), pathways followed, time on task, relationships between events, moments that surprised you, or interactions that aligned with your expectations.
2. Categorize information. You may have some ideas about what you will learn; take note of these themes that you anticipate to develop a catalog of categories. As you review your videos, notes, and/or transcripts, you will notice that participants used words or conceptual phrases to describe the experience – often quite similar ones. Take notes about the ideas, concepts, behaviors, phrases used, etc. for each participant during each task or questions. Then, review this data to add to your catalog of categories. This might require several rounds of review of your notes. Create your own codes to categorize information. Here are examples:

Activity/Prompt Categories What information is most important to you when you plan trips? Budget (B), Arrival Time (AT), One Way versus stops (OW), Date (D), Departure Time (DT), Airline (A), Extra Fees (EF) What, if any, additional information, tools, or features would have helped you to plan the flight? Less Detail (L), Fare Alerts (FA), Calculator (C) 5. Decide what is meaningful. Set up a rubric you will use to determine differentiate between a single occurrence and a pattern (i.e. 1 time is a single occurrence across sessions, 3 times across sessions is a pattern.) Similarly, you will need to determine what is more and less important. This structure enables you to decide the severity rating associated with each issue you find. This rating is based in several factors:

How many times will users encounter the problem – repeatedly as they complete a task or just once. How many users are likely to encounter the problem – everyone or just a few. How impactful the problem is – will it prevent users from completing a task or is it a minor impediment. 6. Develop Themes. Determine which themes are critical. This might be because there is a problem that is blocking someone from completing the task – even though only one person might have encountered it – or because there is an annoyance that every user encountered and remarked upon. This is where judgment is important; sometimes, you will know better than users what the best design approach might be to resolve the problem, and it may not be what they suggest.

1. Finalize findings. Findings might range from recommendations on how to solve problems or additional research questions that require more sessions, or even another test using a different research method. By keeping an open mind about how you might address what you find, you will give yourself an opportunity to come to the appropriate conclusions – and take the right action.

Analyzing data takes time and thought. To that end, giving yourself both active time to think about what you have learned and what it means as well as passive time – time that you might spend engaged in another activity that is unrelated to the project you are thinking about – helps you to come to better conclusions. Research is creative, which means that sometimes, it does not happen on schedule.

Finally, give yourself the tools and space that help you to make associations. For some people, a physical space including with a wall covered with post-it notes and whiteboards is most helpful. Others find that spreadsheets or databases with coded data works well. Talking with stakeholders and colleagues can also bring clarity. Some combination of these approaches might be most effective. Your approach will probably change over time as you do more research, so plan for some flexibility and true different approaches until you find the best fit for you and your team.