Skip main navigation

The impact of racism in social media

In this article, we will explore some ways in which racism in technology plays out in social media, and the influence of social data.
an illustration depicting a social media timeline, in the centre the illustration is a post which features a large exclimation point.
© Creative Computing Institute

Promotion of racist & inflammatory content

Facebook & Youtube’s algorithms promote extreme content, and you might have wondered why this happens.

There are certain behaviours that Social Media sites are optimised for. These behaviours drive metrics, which in turn drive revenue. For example, Instagram users instinctively check the number of Likes in their posts frequently (1).

This behaviour is not coincidental – the platform has thought of ways to encourage it. The more Likes someone gets, the more content they are likely to post, and the more time they and their followers are likely to spend on Instagram.

For the Social Media site, this leads to more opportunities to learn what kind of content and products their users like and therefore more opportunities to advertise relevant content with a higher probability of interest or purchase.

The focus is on advertising

Instagram, Facebook and YouTube sell advertising, so it is in their best interest to design their apps to encourage certain behaviours. The Like button has a lot of influence on revenue. Some other revenue-driving metrics include:

  • How long you spend viewing content
  • Engaging with content (likes, retweets, comments)
  • Sharing content

Inflammatory content tends to drive engagement and therefore drive up these metrics.


Algorithms are adjusted to drive more engagement and therefore content that is inflammatory including racist, sexist or other prejudiced content is disseminated quickly and can have far-reaching consequences, influencing the behaviour and views of everyday people.

This is exacerbated by the increasing use of bots (2).

The influence of social data

In 2016, Microsoft created an AI bot on Twitter called Tay (@TayandYou on Twitter) (3). As we learnt earlier, machine learning is a process where the computer system learns from making inferences from data inputs. In this case, the data inputs came from Twitter users who would interact with the bot.

The intention was for Tay to become smarter through interactions it had with Twitter users. Microsoft said, “Tay is an artificially intelligent chatbot developed by Microsoft’s Technology and Research and Bing teams to experiment with and conduct research on conversational understanding.”

Within 24 hours, Tay had been taken offline as it had started tweeting racist, sexist and homophobic comments. This happened because some Twitter users sent racist, sexist and homophobic tweets to the bot (4). Tay had learnt this toxic ‘behaviour’ from its interactions with humans, which demonstrated how AI can amplify the worst of human behaviour.


  1. Janarthanan Balakrishnan & Mark D. Griffiths, 2019. An exploratory study of “selfitis” and the development of the selfitis behaviour scale, NCBI.
  2. Massimo Stella, Emilio Ferrara & Manlio De Domenico, 2018. Bots increase exposure to negative and inflammatory content in online social systems, PNAS.
  3. Peter Lee, 2016. Learning from Tay’s introduction – The Official Microsoft Blog, Microsoft.
  4. Oscar Schwartz, 2019.In 2016, Microsoft’s racist chatbot revealed the dangers of online conversation, IEEE Spectrum.

Further resources:

  1. Alastair Reed, Joe Whittaker, Fabio Votta and Seán Looney, 2019. Radical Filter Bubbles, Royal United Services Institute for Defence and Security Studies.
© Creative Computing Institute
This article is from the free online

Anti-Racist Approaches in Technology

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education