Limitations to the use of AI

In the previous activity, you have learned about the potential of artificial intelligence. Although this potential is very promising for the future of healthcare, it also has its limitations which have to be considered before implementation. In this step, we would like to introduce some of the limitations of artificial intelligence in general and in healthcare in particular.

Despite the promising potential of AI-based technologies, there are significant limitations associated with their applications in medicine. Let’s first discuss three arguments why AI cannot replace all tasks of the physician.

AI cannot replace all tasks of the physician.

1) Artificial narrow intelligence vs artificial general intelligence in medicine

AI-based systems are becoming more efficient than human professionals in performing certain routine tasks, such as the identification of malignant tumors on X-ray images. AI systems that are able to perform one particular task or the combination of such individual tasks are known as artificial narrow intelligence (also known as narrow AI or weak AI). These systems might be able to solve that one task very well, even better than humans. However, they will fail in executing other tasks that they have not been trained for. For example, an AI model trained to identify breast cancer tumors is not able to process patient history and prescribe medication. On the other hand, artificial general intelligence (general AI or strong AI) systems are thought of being able to perform the same intellectual tasks as humans, learn and transfer knowledge between different domains. However, currently this is only a theoretical category and what you can meet in everyday life is only narrow artificial intelligence. Thus, AI systems cannot replace all complex tasks that doctors perform on a daily basis.

2) Human in the decision loop

Further, even though some algorithms appear to outperform clinicians in an execution of a particular task, it is currently necessary that a physician evaluates the result of the algorithm in each of these tasks. To this date, physicians perform more various tasks combined than available individual AI algorithms. Moreover, unlike AI algorithms that require lots of specific data for training, human professionals are able to make decisions even when only limited data is available. In order to trust the technology there should be a possibility to interfere with or cancel the decision made by the AI system. This defines the role of AI as a support tool to improve diagnostics, decision-making process and clinical care opposed to substitution of healthcare professionals.

3) Communication with the patient and empathy

There are many AI limitations in the clinic, but one of the most important is the communication of medical visit results, selected treatment or procedure to the patient. In order to convey results to a patient appropriately, a doctor needs to use empathy in this case. At times the results are quite sensitive and could involve a life-changing diagnosis. To date, AI cannot convey results in an empathetic way. Therefore, it can only assist in outputting results, while the physician still needs to be the one communicating to the patient. Additionally, a black-box nature of AI-based systems may lead to the lack of trust from the patient. A trained healthcare professional will be able to interpret the results and convey them to the patients in a trustworthy and easily understandable fashion.

Even with narrow AI systems that are built with the purpose of supporting physicians in their daily tasks, obstacles are encountered that prevent the adoption of such systems in clinical practice. Despite the growing number of publications demonstrating the promising use of such systems, only few have been evaluated in a clinical setup. This is largely associated with the level of readiness of the technologies for the implementation in patient care.

Readiness of the technology

One major drawback in artificial intelligence is the reliance on training data and the lack of generalizability to other data. AI algorithms are often developed by training on data from one specific hospital or clinic. This algorithm could show a very high performance on the test data from that same hospital. However, when the resulting model is used for data from other hospitals, the system probably does not work as well. The algorithm needs to be retrained or re-calibrated using this new data. It is therefore critical that new algorithms are validated against completely new datasets that are independent from the original algorithm. Notably, algorithms should also be tested in randomized controlled trials, something that is currently exceedingly rare.

This type of preference towards one type of data on which the data has been trained is called bias. Bias can also emerge as a consequence of time. That is, if the data has been generated in the last decade or further in the past and demographic shifts have occurred since then, this will also affect the efficacy and accuracy of the system on new data. This also introduces the need for continuous re-training of the AI algorithms. However, there is practically no healthcare organization that has the data infrastructure and skillset to conduct validation, deployment, and continuous re-training of AI algorithms.

An AI model can also be biased towards the content of the data itself. For example, in the Netflix movie Coded Bias, MIT researcher Joy Buolamwini showed that facial recognition technologies were not able to recognize her face, due to her dark complexion, until she put on a white mask. The reason for this was that these technologies were only trained on example data containing faces with light or fair complexion.

There are many other types of bias that can affect the effectiveness of AI systems in realistic situations, which will be discussed in more detail next week. Bias is one important reason for reluctance towards the use of AI in healthcare, especially in combination with the black-box nature of machine learning and in particular deep learning.

You have learned before that machine learning algorithms are trained by looking at example data and using those examples to establish its internal connections (such as the weights that connect the different layers of a neural network). However, the only thing that the user sees in the end is the output, i.e. a prediction made by the algorithm. What happens in between is hard to be visualized, especially in deep learning networks, which is why this is considered a ‘black box’.

This lack of transparency of how the algorithm makes its predictions makes it harder to trust and accept the predictions that are being made. For that reason, we have to think of ways to make AI explainable. This is a very relevant issue in the world of data science in general and several techniques are being developed to make this happen, but it is still a work in progress.

These and other challenges (e.g., logistic issues concerning the implementation of AI in the clinical pipeline) will be discussed thoroughly in Week 2 of this course.