Shape completion using Gaussian Process regression

Welcome back. The goal in this tutorial video is to learn how to perform Gaussian Process regression in Scalismo when given correspondences and also to learn how to use this regression to complete shapes with a missing part. The shape that we will be completing today is the nose of our boss, Thomas Vetter, that you previously saw disappear in the video. Let’s start by loading our noseless face, or face with a missing nose and display to the scene. Here you see this face with a hole instead of the position of the nose. And since we now intend to complete this missing nose using our face model, let’s also load and display our shape model.

And maybe let me colour the model such that we can easily identify it. So you see here that we have an instance, or the model mean here, is an instance that is quite far away from our target mesh. The idea of Gaussian Process regression is actually that we’re interested in finding now an instance out of our model that is similar enough to this target face such that then we can simply take the nose away from this fit and then substitute it to our target face. You saw in previous videos that in order to perform a Gaussian Process regression we actually need correspondences.

What we can do now is since we don’t have any yet, we can start by clicking some correspondences. So I will now start by clicking a few characteristic points, such as for example here the corners of the eyes and this point on the chin. And then do the same now for my target mesh.

So now that we clicked our landmarks in the user interface, we can actually retrieve their position programmatically. And this we can do using this getLandmarksOf method where I specify as a parameter the name of the object in the scene. So if you remember, we named our model here, model, and also our mesh noseless. And these are now the parameters that we use for retrieving the clicked landmarks in the scene. Then, once we have the list of landmarks we just get, for every landmark, the point position of this landmark. And this is why we have a sequence of 3D points. I do this now for both the model points and the target mesh points.

This is now why I have my list of points. Having these correspondences in the Gaussian Process regression means that we actually observe a part of the deformation field. And we would like now to use our model and this regression feature of Gaussian Process to actually retrieve the full deformation field that fits to the observed data and still obeys to the properties of our model. So this means that it should still give us a valid face. So let’s start here by displaying, first of all, this partial deformation field that we observe. So what I’m doing here is again, as we did before, I’m building a discrete vector field that is the defined over a domain.

There are the points here that are the model points that I clicked before, and then, with associated vector values that are the deformations. Here, these are the vectors that are pointing from the model points to the points on the target face that we just clicked before. If I now have a look at this deformation in the scene and maybe make this face invisible, you see that, what we observe here is that with these correspondences we have deformation vectors pointing from the model points to the target points for a few points of our model. And what we would like now with the Gaussian Process regression is to find again this full deformation field.

The way we do this, or one way of doing this in Scalismo, is actually using the posterior method here. That is a method of both Gaussian Processes and also of statistical mesh models. And what this method takes as a parameter is a sequence here of training data. And this sequence is actually a sequence of triplets. These triplets are, first of all, the point identifier of the points for which we actually observed the position. So these would now be the identifiers of these blue points on the mean mesh. Then for every one of these identifiers we also specify where we actually observed the position of this point.

So in this scenario this would be the position of these pink points that we clicked on the target mesh. In addition, the third element is actually a noise element for the training data that says how certain we are about every observation that we clicked, that we noticed. Here, I chose to always have the same noise that is a normal distribution that is a multivariate normal distribution, which is an n-dimensional normal distribution in Scalismo, that has now a zero mean and diagonal covariance matrix with a half a millimetre variance that you see here which basically says that I’m rather sure of the landmarks that I clicked of correspondences that I just clicked.

So if I run this and display it in the scene, you see that now I get actually a fit, a model fit, so this is now the mean of my posterior. But this is not a single mesh. It’s actually a space of meshes that satisfy my observations. And you can really verify this for yourself if I sampled randomly. You see that the characteristic points that I clicked, which are the corners of the eyes and this point on the chin, they tend to remain around the position of the pink points that I specified and this up to the noise parameter that I specified as well in the GP regression.

So hopefully you see that by clicking a few correspondences we start to have model instances that are similar to our target face. But we’re not quite there yet. The question is now, what if we had more correspondences. Well here I prepared a few correspondences that I store in this file here called modelLandmarks.json that I can then read using the LandmarkIO object here and add it to the model. I also do the same now for more than 200 points actually, that are corresponding to these landmarks and they are also defined over the target mesh that is our noseless face. And I also now read it and add it to the scene.

And if I now make the meshes invisible, and maybe clean also this deformation field here, you see that now we have many more correspondences between the blue points that are defined over our model and the pink ones are now defined over our noseless face as you can verify it here. And these are uniformly sampled points over the entire face. So now that I have more correspondences I can do exactly the same operation that I did before, simply now using more points. So now I’m forming this training data index sequence exactly with the same concept as we did before, the same procedure as we did before.

So first, specifying the point identifier that identifies the blue points, then the corresponding pink points here, and using the same noise that we used before, and now also computing the posterior model and displaying it in the scene. And as you can see here, now we get a new posterior model that actually starts to resemble much more our target face. And again this is now a normal distribution over faces from which we can now sample.

And you notice that all of the samples actually are rather similar to the target face and we can now finally select or take away this nose part out of an instance, let’s say of this posterior model, and then try to fit it to our target mesh. So now that we have our posterior model that actually fits well to our target mesh and since we’re interested in deformations only of the nose, or actually retrieving only the nose, what we can do now is actually marginalise our posterior model on the nose only.

And this we can do in a similar way to what we did in a previous video. So we start first by identifying the point identifiers of the points that happen to be on the nose and here, what we do is, we simply filter over the point identifiers by retrieving points that are within 42 millimetres of a point that happens to be somewhere in the middle of the face. And then given these nose point identifiers we just marginalise our posterior model specifying again this list of identifiers. And what we get back is a triangle, a statistical mesh model in Scalismo that we then display in the scene after cleaning it a bit.

So you see now we have a marginal statistical model over our posterior model that actually fits to our face, and if we now sample from this nose model, you see that we get sample noses that are rather similar or very similar to each other that actually fit well, would fit well to the target face. In fact, if we now display the target face, noseless face here, you see that we actually have a rather nice fit of our posterior model to the marginal of the posterior model to this target face. And if I now go back to the mean of this model this would be now the instance of our model that is the most probable according to the observed data.

And this would be, in the statistical way, the nose reconstruction that we would recommend. But now that we have also a statistical model, what we could do is, we can play in the parameters of our model and then change, like for example here, the coefficients of the eigenmodes and then we can actually obtain different similar noses from this model. And we can also now sample different random samples or take different random noses that would actually still fit to the observed face.

So here we are. We saved the day, and we gave our boss back his nose. So I encourage you to go to the companion document to this tutorial and give the reconstruction a try for yourself.