Skip to 0 minutes and 9 seconds Hello everyone! And thank you very much for watching this video. In the next few minutes we’ll be talking about reinforcement learning in healthcare and I’ll be discussing some of the opportunities and challenges associated with this new field of research. If we want to take a step back and if we were to define what machine learning is really what this is about is about learning from the data. and there’s really three pillars in machine learning. The first one being supervised learning, and then you have unsupervised learning, and the third aspect is reinforcement learning which will which we’ll be talking about in a bit more detail. Supervised learning.
Skip to 0 minutes and 48 seconds if we were to define this, it’s about learning the relationship between input data and an output data so what we took, what we call learning a function between between x and y and within supervised learning. Two really famous subtypes of supervised learning are regression and classification algorithms and all we give you; I will be giving you an example in a few seconds. The second aspect is unsupervised learning where we learn about the structure in the data so this is called unsupervised because we have no idea but the data is unlabeled.
Skip to 1 minute and 31 seconds We don’t need to know the specific categories of data before they are fed into the algorithm and in unsupervised learning - really well used types of algorithms are clustering and dimensionality reduction. We won’t be going into any detail about this and the third aspect is reinforcement learning where this is a bit different in the sense that what we want to do is learn an optimal strategy. Let’s talk about an example of supervised learning to start with. Of course, I chose an example that comes from the field of healthcare and this was a very groundbreaking paper that was published in 2017.
Skip to 2 minutes and 16 seconds In the journal Nature showing that a computer system was able to diagnose skin lesions with at least as good accuracy as the the most experienced dermatologists. So this this is an example of supervised learning where the the the data used for training the model is labeled meaning that we know which lesions were cancerous which lesions were non cancerous rent when when training the model. But let’s jump straight into reinforcement learning the intuition behind reinforcement learning is that what we want to do is learn an optimal strategy. And it’s often I would argue… it’s often more complex than prediction tasks such as the one we we saw in a previous example, about skin lesions.
Skip to 3 minutes and 11 seconds The idea behind the reinforcement learning and here is here is a very simplistic example if you think about a rat in a maze, this example we simply define some of the elements that are necessary in order to implement reinforce planning algorithms so in this example think about a rat in a maze the rat is the agent; the maze is the environment and the agent can be in in lots of different states that are represented by the cells in this maze. In every state of the system that the agent has the choice of making different actions can go up left, right or down.
Skip to 3 minutes and 50 seconds and the objective of the agent is to maximize the reward to reach cells associated with the reward as often as possible and to avoid cells associated with a penalty as often as possible and the actions that lead to reward are reinforced hence the term reinforcement and if we take a very very simplistic approach this this could be put in parallel with with healthcare where we have a patient, the patient can be in various health states and various clinical conditions.
Skip to 4 minutes and 27 seconds And the objective of physicians and the healthcare system in this example is to take the patient from state to state by intervening on our patient and with the objective of bringing this patient home rather than sending him to the cemetery, so again in this example we have different states and the the agent the physician has different options of actions and interventions in every state of the system and the objective of the doctor is to send the patient home to maximize the reward if survival is the reward. Taking a step back we can think about how do doctors achieve this.
Skip to 5 minutes and 16 seconds So typically, nowadays when a patient comes in the hospital with a with a particular presentation what doctors do is that they will be gathering data about this new patient examining them doing all sorts of assessment and examinations. Then they will use their theoretical medical knowledge as well as their clinical experience meaning all the cases they have previously encountered. All this information will be integrated and will lead to a medical decision. The problem is that this is a very very suboptimal approach for various reasons. A very important aspect is that humans are are full of biases and cognitive biases in medical decisions is a very significant problem.
Skip to 6 minutes and 5 seconds For many diseases, many clinical problems we don’t have a physiological model or sometimes we lack theoretical knowledge so… personally I’m an intensive care doctor I don’t pretend to have all the all the medical knowledge about all the medical specialties in the world for sure Also human doctors are not perfect they sometimes forget cases that they have encountered in the past sometimes the cases are so rare that very few people can even make a diagnosis sometimes we get the diagnosis wrong and practice variation is also a huge problem
Re-inforcement Learning in Healthcare: Opportunities and Challenges
An experienced anaesthetist and intensivist with 10 years in training and practice, Dr. Matthieu Komorowski holds a full specialist board certification in both France and the UK. He belongs to the MIT Critical Data group where he applied machine learning to large critical care databases, with the objective of developing decision support systems for sepsis, the number one killer in intensive care. In this video, he explains re-inforcement learning in healthcare.