Skip main navigation

The risks of AI

Learn about some of the risks associated with AI.

This week, we have spent a lot of time thinking about the various risks involved in user research, and how we can minimise them through careful planning and preparation. AI technologies can be very beneficial, but new technologies also mean a new set of risks that must be addressed.

Bias

As we learned in step 2.10, bias is a common challenge in user research. We are all human, and are susceptible to preferences, beliefs, or prejudices that influence the way we view and interpret the world around us.

When we collect and analyse data, we may do so in a biased way, by prioritising certain groups over others. For example, marginalised groups are historically underrepresented in healthcare studies, meaning that a large amount of healthcare data is skewed towards certain social groups.

AI models are often trained on historical data sets. If these contain bias – even if it was unconscious or unintentional – an AI model designed to identify patterns and make predictions, or generate new content based on data it is trained on, may replicate these biases and produce distorted outputs. This can have serious consequences if skewed user research data is used to inform policy or design decisions for government services.

In government, you should only ever use AI tools that have been tested and approved by your department, or that have been developed in-house for specific purposes. However, even with approved tools, the quality of data you provide is critical. Bias can creep in, even when data is collected specifically for training purposes. Minimising the risk requires conscious effort.

As a user researcher, it is your responsibility to reduce bias during data collection. This is essential for all user research, but the risk can be amplified when using AI tools. AI will act as it has been programmed to act, so biased, incorrect, or incomplete data will lead to biased or incorrect outputs.

Hallucinations and mistakes

One of the biggest risks of using AI is a phenomenon called hallucination. In simple terms, AI works by identifying patterns in data and using sophisticated guesswork to generate outputs based on those patterns. These outputs are essentially based on probability – generative AI will produce a likely output for a given input based on the patterns it learned from the data it was trained on. It does not “understand” the content in the way we do.

This means that it sometimes produces outputs that appear plausible, but are factually incorrect, known as hallucinations – outputs which the AI has essentially “imagined.” This is a significant risk for AI used in user research.

For example, let’s go back to user personas, which we learned about in step 1.8. User personas are representations of user groups that share certain characteristics or attitudes. We identify these shared characteristics or attitudes through analysis of our user research data; creating personas that accurately represent different user groups and types.

AI tools could definitely speed up this process, especially when working with large datasets. The AI might even spot themes or patterns that humans could miss while trying to analyse such a large volume of information.

However, caution is needed. An AI-generated user persona could seem plausible, but it could be the result of a hallucination, such as a pattern it has imagined or a mistake in the data it was given. If the error is not spotted, it could then be carried forward into the service design process. The team could find themselves trying to build a service and solve problems for users that do not actually exist. A persona generated with AI could also reflect bias, favouring certain characteristics or groups and excluding others. AI tools can help us to work efficiently, but it is essential to check outputs carefully to ensure accuracy. Mistakes that go unnoticed can be costly and difficult to fix later on.

Ethics and data security

As referenced in the AI Playbook for UK Government, the ethical and responsible use of AI in government is crucial for:

  • maintaining public trust
  • protecting individual rights
  • fostering equitable societal progress

This is equally true for AI in user research. In step 2.12, we learned that data protection is not just about compliance with the law – it is about integrity, respect, and trust.

Participants share sensitive and personal information with user researchers, trusting that it will be handled responsibly. The rise of AI has introduced new challenges and concerns for data ethics and protection. Many AI tools are owned and operated by private-sector companies that are based outside the UK, which raises concerns about how the data is stored, processed, and used. Earlier, we explained that you should only ever use specific tools that have been tested and approved for your user research. Even then, you must take extra care when working with sensitive data to avoid breaches of confidentiality.

To give a recent example, in May 2025, civil servants working for the Scottish government were told that they should avoid discussing policy with commercial AI agents, such as ChatGPT. These tools ‘learn’ from user inputs, meaning that sensitive data could reappear in future outputs – potentially visible to anyone.

When it comes to AI and data, you also need to consider the human connection that can be lost if you rely on AI to conduct your analysis. This is another ethical consideration to how AI interprets data. AI agents are not able to respond sensitively to traumatic or distressing content, and, as we explained above, are susceptible to biases that could have harmful outcomes for marginalised user groups.

AI tools are just that – tools. They cannot replace human interaction or interpretation. As a user researcher, you have a duty of care to your participants, to protect their data, use it responsibly and ethically, and treat them with empathy and respect.

What this means for your practice

AI technologies are evolving rapidly. They offer exciting possibilities for user research, but they also introduce new risks, from bias and hallucinations to ethical concerns and data security.

As user researchers working in government, we must approach AI with care, curiosity, and critical thinking. These tools can support our work, but they cannot replace the judgement, empathy, and responsibility that lie at the heart of good user research.

Task

Even if you do not personally use AI, it is important to be aware of the benefits and risks of this technology. Let’s think more specifically about the potential risks of AI in user research.

Consider the following questions:

  • are there areas where you feel AI might limit your development or understanding as a user researcher?
  • could relying on AI for tasks like analysis, writing, or synthesis mean we lose vital skills – or does it simply change how we apply them?
  • what additional steps might you have to take if using AI tools in research (thinking about data protection, for example)?

Share and discuss your thoughts with your fellow learners in the comments below.

This article is from the free online

User Research in Government: Developing Further

Created by
FutureLearn - Learning For Life