Model fitting and correspondence

Welcome to Week 6 of the shape modelling course. This week marks the start of a new topic. We will shift our focus from building shape models to actually applying shape models for analysing data.

A shape model gives us statistical knowledge about how a shape looks and varies. It can be used to answer questions like, is this shape normal? Does it maybe show pathologies? And how would it look like without the pathologies? So what I would like to use it for is to analyse data, such as this image. But to be able to analyse the data, I somehow need a way to compare the information that is encoded in the model with the information in the image. Now if you think back, what actually a model represents, it is probabilistic information about deformation fields. So we built a model by selecting a reference shape and then modelling deformation fields using a Gaussian Process.

If I want to answer such questions, I would somehow need to know how does the reference relate to the data that I’ve seen. So I would, for example for this image, need to know what is the deformation field that brings my reference to this shape that I see in the image.

For practical application, there are two representations of data that are of relevance. One is a representation as a surface or in 2D contour. And in many practical application, we actually have images of data sets. And we want to analyse these images.

In this week, we will focus mainly or purely on surfaces and contours. We will develop a method how we can fit a model to the contour and the same method can be extended such that we can use it for images. And this will be the topic of next week. The setting is as follows. We start with a reference shape and a target shape. And what we would want to do is we try to find the deformation that maps this reference shape to this target shape. Now all the information that is in the model is only about shape deformations. It doesn’t include rotation. It doesn’t include translation.

So the way the target shape is now, I cannot use it directly to find the deformation. So the first thing I have to do is I need to align the two shapes. There are many methods to do that and one you have learned about in this course, which is Procrustes Analysis. So now let’s assume the two shapes are already aligned. So how can we find a deformation from the reference to the target?

A very naive and simple idea would be to say, OK, I just choose a number of points of my reference and then I try to find the closest point in the target. And you see the result here on the left. It actually defines a deformation from the reference to the target. But it is not really what we would like it to be. So for example if we just considered this point here, the tip of the thumb of the reference, it is not matched to the tip of the thumb of the target, but rather here to the side. Another problem is this deformation has jumped here. So it’s not smooth.

What we see is this deformation, it defines the deformation but not correspondence. It doesn’t map to the points that we would like and it doesn’t have the properties we need and not the properties of deformations that we modelled.

The reason is that this closest point assumption is actually too weak, so we cannot just assume that a deformation is defined by its closest point. We should assume much more about the deformation that relates the reference to the target. For example, one thing that I can assume is that the deformation should be smooth.

I could even assume more. Maybe I can assume that the results of such a deformation looks like a hand because here I already know that we’re working with hand shapes.

And actually the smoothness and the assumption that the result looks like a hand, this is exactly what is incorporated in our shape model. So the question could be, could I maybe use directly the shape model that we’ve built for analysing the data set? Or in other words, could I fit the model to this data set?

Whenever we talk about fitting a model to a data set, the first thing we should think of is Bayes’ theorem. So Bayes’ theorem gives us a very principled way to use prior knowledge that is given in probabilistic form. So Bayes’ theorem is essentially a simple formula. It tells us if we have some prior knowledge, plus a likelihood term. And I divide that by some normalisation term. Then I get here this probability of u given the data. And this probability of u, if u is a deformation field, this would in our case be the probability of a deformation given the data. This is exactly what we want.

In our case, the prior knowledge is just the shape model and the likelihood function would be something like how close will my deformation field u bring me to the target? Unfortunately, while this approach is nice and principled it’s also a bit complicated to get right. And in this course, there is not enough time for discussing that in details. We will therefore discuss a little bit a simpler solution, which is a very popular heuristic. The heuristic is called the Iterative Closest Point Algorithm. It is a classical algorithm, which was proposed by Besl and McKay already in 1992 with the aim of minimising the distance between two point sets. The idea is very simple.

They start by finding closest point between the target and the reference. This is exactly what we did. But then, as for us, their correspondence or the closest point might not actually lead to a good solution. But the idea is that they can use steps nevertheless to estimate a transformation based on these corresponding points. And then they transform the reference using this transformation and they iterate. They find the closest point again and they hope that this time it is a little bit better. In their case, the transformation they considered was actually rotation, translation, and scaling. For us, we will adapt it a bit such that it works with Gaussian Process regression.

So let’s see if you can use that. So we want to include prior knowledge. That means we want to use our shape model. So we start finding the closest point directly from the mean shape that we have given in our shape model. And from this mean shape, we find the closest points to the target.

Given these closest point, I can compute the deformation from the closest point on the target to the corresponding point on the reference. This gives me and deformation field. As I see here and maybe here and also at some other point, it is still not very good. What we hope, however, is that it’s good enough that it can be used for a Gaussian Process regression procedure. So now we want to include the prior knowledge. We want to include everything that we know about hand shapes. And the way to do that is we perform a Gaussian Process regression, treating these deformations as noisy observations. And this gives us a new model with a new mean.

And we hope that that mean is actually closer. And this is exactly what happens. So in iteration 2, I get this new mean shape that we see here in grey. We see that it’s quite close. And now I can find the closest point again, which gives me a better solution.

Now the defamation field, if I iterate that, is already a bit better. Still, it is not perfect. It has some problems at some points. But I can take the same idea again, perform a Gaussian Process regression, and use the deformations as noisy observations.

If I do that around 20 times, then this is what my mean shape looks like. We’re almost there. Finding the closest points from this shape to the target shape is actually fairly easy. And the final deformation field, I get after 20 iteration, that looks like this here. We see now this procedure actually established correspondence. So the tip of the thumb is mapped to the tip of the thumb. And if I would investigate that closely, I would have very reasonable correspondences. And now that I have done that, I can actually use all the information that I’ve given in a shape model to analyse this target surface. I could try to determine whether it is pathological or whether everything is normal.

Or I could try to reconstruct a given pathology.

You may remember from Week 2 that we still had the open problem when we wanted to learn a shape model from data, how can we find the deformation fields that relate to reference to the example data sets that we wanted to use for learning. It turns out that this procedure can actually be adapted and used exactly for that purpose. You will find details about that in the next article.