Model fitting with Iterative Closest Points

Welcome back. The goal in this tutorial video is to learn how to use the Iterative Closest Point method to fit a statistical model to a target mesh, and also retrieve correspondences while doing so. So let’s start immediately by loading a target mesh that we would like to fit in our model, and display it in the scene. And here, we see this familiar face now that you know, with its original nose. And since we know we want to use also our face model to actually fit to this target, we can now also load our model and display it in the scene.

And as you can see here, now we have our model that is loaded behind the target face, and we have an instance of the model that is the mean face that is still far away from our target mesh. And the goal in this video now is to try to find an instance out of this model that is similar to our target, and in this case, without having correspondences.

So since we are searching for correspondences, I here selected or preselected a few characteristic points that are easily identifiable, that I store here in the file called icpLandmarks.json. And I load them using the LandmarkIO object. And then I simply add them to the model in the scene. And you see it here that now we have these landmarks that are added that are at points that are easily identifiable. So we have points at the for example corner of the eyes, at the tip of the nose, the corners of the mouth, and also here, easily identifiable points on the ears. And the goal now is actually to try to find correspondences for these points on our target mesh.

So the question now is, how can we find these correspondences? Well, the simple way the Iterative Closest Point method suggests is actually to attribute the closest point as candidate correspondences. So if we now go back to our scene and visualise the position of our clicked points, these blue ones that are clicked on the mean of the face and visualise their position with regard to our target mesh, what the Iterative Closest Point method would suggest is for every one of these blue points now to attribute a candidate correspondence, that is now the closest point on this mesh to this point here, to this blue point here.

And if we now try to translate this method into code, then we can do this as I do here, by first defining this attribute correspondences method. So this is now a method that takes as argument a sequence of 3D points, and these are now the blue points that we clicked on the model. And then we transmit a sequence also of corresponding points that are now points that will be, or should be points that are on the target mesh.

And the way this works is by simply now looping over the sequence of given points, and then for every one finding the closest point on the target mesh– so this is done by calling this findClosestPoint method, that is a method of the triangle mesh in Scalismo, and then retrieving the point position of this closest point. And what I did here is then simply apply this method to the blue points, the characteristic points that I loaded from file, and then display their corresponding points, according to this rather simple method.

And if I now go back to the scene and colour these suggested correspondences as you can see here– So you see that to every one of these points here, we found again this candidate correspondence that is the closest point. And if I try to see where these points do actually lie on the target mesh, you see that we have a more or less OK initialisation for our correspondence, so a few points are more or less at the good positions, like for example the corners of the mouth here, the corner of the eye, for example. But if you now look at the tip of the nose here, for example, we’re still a bit far away.

And also here, for example, at the ear area, we still have quite some work to do. So this is an OK initialisation with this basic method, but we’re still far away from the correspondences that we would like to have. Now, the idea behind this Iterative Closest Point method is to say even though we don’t have such perfect correspondences, we nevertheless can use them in a Gaussian Process regression method. And hopefully, the idea is that when we do that, we actually get corresponding points to our blue points that are in the model fit that are hopefully at the better position than these candidates correspondences.

So before we do that, I would like first to visualise this deformation field, or partial deformation field that we’re going to be feeding to the Gaussian Process regression method. So what I’m doing here is again, as we did quite often is I’m building a discrete vector field that is defined over the domain of the model points, and with values here that are the vectors pointing from the blue points to their candidate correspondences that are found with this closest point method. And if I now visualise this deformation field, you see that this is now the partial deformation that we’re going to be feeding for our Gaussian Process regression, based on these again candidate correspondences.

So now that we have our partial deformation field, we can plug it into our Gaussian Process regression framework using the posterior method, as we did before. And what I do here is I define now this fit model method that you see here, because I intend now to be calling this regression face iteratively, in the later phase of this tutorial. What this method now here, fit model does is it takes as a first argument now a sequence of point identifiers, and these are now the identifiers of the blue points that you see here. So these are the identifiers of these points on the mean mesh of our model.

And it also takes as a second argument a set of point positions, 3D point positions, that are the candidate correspondences that we observed, and these would be in this case these orange point positions that you see here. And what we do then is we simply build our training data, as we saw previously, which is again this triplet consisting of a point identifier that is coming from our given list, the corresponding point position that is observed that is also given to us in this candidate correspondences, and then also this noise parameter that specifies the certainty. And here I define the nose parameter up, here with a one millimetre variance on the diagonal.

And then once we have this training data, we again call this posterior method to obtain a posterior model. And the special thing about this method here is that instead of returning the posterior, now we are just returning the mean of this posterior, because we’re actually interested in the mesh, or the most probable mesh according to this observed data. And what I do is I simply now apply this method to the identifiers of the points that I loaded from file. So these are, again– the identifiers are these blue points on the model, and the candidate correspondences that we found with this find closest point method that we applied before, and then simply displaying the result of this fit in the scene.

So if we now look in the 3D scene at our fit that resulted from this observed deformation field, and especially if you observe now the position of these orange points on this target fit, you notice that they are not quite exactly at the characteristic points that we chose, right? So the tip of the nose, for example, is not exactly where it should be. And this is because the model actually didn’t exactly fit the candidate correspondences that we proposed, but instead has got close to these positions.

And if we in fact now want to see where the actual corresponding points to our blue points of our model are in this fit, then we can execute now this part of the code here, where simply for the point identifiers of these blue points that I had before, I simply loop on them and then for every identifier I locate the 3D position of this identifier using the point method of the triangle mesh class. So I locate it on this fit, and then I simply display the set of points there. And if I now go back to the 3D scene, you see that these points now are again where they actually should be. So let’s colour them maybe pink.

So these are again at the points that are the characteristic positions for which we want to find the correspondences. And if we now have a look again at their position with regard to our target mesh and look from the inside here, you see that we actually got much closer now to the target mesh. And again, the difference here with regard to the previous situation that we had with the orange points is that these are now real correspondences. These are no longer candidate correspondences, but real correspondences given by our model. And now we have a much better starting point that is closer to our model.

So if we were now to iterate this whole operation again, we would have a much better starting point. And hopefully, we can find better candidate correspondences. So let’s now see what happens when we do this same procedure repeatedly. What I’m doing here is I’m defining a recursive function, called here recursion, that at every iteration takes two parameters. Here it takes first this number of remaining iterations, and it also takes a current positions of the points based on which we would like to do a fitting.

And for example here, in the first iteration of when we call this function recursively, we would typically feed it the positions of these blue points here that are the initial positions of the points for which we would like to find correspondences in our model. And now that we have these current positions, what we do is we do what we did– pretty much the same thing that we did previously. So we first find candidat correspondences using this attribute correspondence method that if you remember now finds closest points to these points on the target mesh. And then we perform a Gaussian Process regression based on these candidate correspondences, which gives us a triangle mesh or a candidate fit out of the model.

And then based on this fit, we’re actually now retrieving the new position of our characteristic points on this fit, as we did before. And this would be now, for example, the equivalent of these pink points that we displayed before on our fitted mesh, except that now we’re doing this repeatedly, so once at every iteration, and we’re actually displaying now the position of these points at every iteration. And then if I have still iterations left, then here, first of all, I sleep in the thread, in order for us to be able to follow in the 3D scene what happens. And then I call this recursive function again, with now this new position of the points.

So now in this case here, I would call now the recursion with this new position where the past current points would be now the position of these pink points that you see here. So let’s now run our recursion for five iterations, and starting from the mean point from the loaded landmark positions, the initial blue points, and see what happens in the 3D scene. So if we follow now this point here that is close to the ear, you see that at every iteration of our recursion we are actually getting corresponding points that are getting closer and closer to the corresponding point on the target mesh, which should be more or less somewhere in this region here.

So of course, we might not necessarily reach it with this little number of candidate correspondences, and also with this little number of iterations. But you see here, hopefully, this gives you the intuition that with this iterative method we’re actually heading in the right direction. And in fact, if I now display here, you see that in red, the position of the final iteration. So this would be iteration five. You see that actually, also here we did move more or less, or slightly towards the right direction. So let’s now do this same, exactly the same operations, with now much more points.

So what I’m doing here now is instead of taking these characteristic few points, I’m now sampling uniformly on my mean mesh 5,000 point positions. And I’m taking the 3D coordinates of these points, and then also now displaying them in the scene. And if I go back to the scene and maybe clean up a bit–

You see that now I have sampled uniformly these points that actually these are not points from the sampled ones. So the points that are actually on the mean mesh, these are now the new sample points for which I would like now to perform an Iterative Closest Point method. And what I’m doing now is I’m doing exactly the same operation as we did before. So first, identifying or retrieving the point identifiers for these points, and then defining a recursive method to perform this Iterative Closest Point fitting, that again has pretty much the same operation as the previous recursions.

So we actually find the closest point to every one of these 5,000 points, and then do a model fitting based on this partial deformation field that we get. The only difference here is that now instead of following the position of the new corresponding points, I’m now instead displaying the fit that I’m getting at every step, which is now the mean of the posterior that I get at every iteration of this iterative closest point. And then once I define now this recursion here, I can now run it, again starting from the initially sampled points, so the initial 5,000 points that I sampled on my mean, and now for 10 iterations.

So now before we run this recursion for 10 iterations on our target, let’s first start by colouring this target mesh in our scene, such as we can more easily track what is happening during the iterations.

And now, let’s run again now this recursive method that we defined, again, based on the initial sample point, and now for 10 iterations.

Let’s see the results. So here you will see that at every iteration, we actually get now a model instance that is getting closer and closer to our target mesh. And you see that at every iteration, pretty much the same way that every point was adapting and getting into better position, here all of our 5,000 points that we sampled are getting to better and better positions, which ultimately yields a better fit of our model to this target mesh.

So you see that we get a very, very similar instance of our model to our target, as you can guess somehow by this overlaying of the colours that you see here in the meshes.

So you see that using this simple and intuitive Iterative Closest Point method, we manage actually to fit our statistical model to the target mesh, and also obtain correspondences by doing so. I now encourage you to check the companion document to this tutorial video, and give it a try for yourself.