Shape modelling with Gaussian Processes and kernels

Welcome back. The goal in this tutorial video is to learn how to build continuous Gaussian Processes from generic kernels and not only sample covariance kernels, and also to experiment with different parameters of these kernels and also combinations of them. So what we know from Gaussian Process theory is that in order to build a Gaussian Process or a continuous Gaussian Process, all we need to define is actually a mean function and a covariance function. So let’s have a look how we define the mean function in Scalismo. What we do here is we want to define a zero mean function. So this is a continuous vector-valued function that at every point returns us a null deformation.

And to do this we define we use the vector field class constructor. And vector field is, in Scalismo, the class of continuous deformation fields. And we feed to this constructor first the domain on which the function is defined. And in this case, we chose a function to be defined on the entire 3D space. And we also pass in as a second argument the actual function that applies to every point of this domain. And here, since we are interested in a zero-valued vector field to every point pt, we actually associate a null deformation vector. There is a 3-dimensional vector with zero components.

The second element that we need to define our Gaussian Process is, of course, the covariance function that is also known as the kernel. In this case, we are interested in building a Gaussian Process using the Gaussian kernel, or squared exponential kernel, that is defined under this formula here. And to do this, what we can do in Scalismo is we actually call the Gaussian kernel constructor here by specifying the two parameters l and s that you see in the formula. So what we do is we first specify for the constructor the standard deviation l. And then once we have our Gaussian kernel, we scale it by this parameter s.

The return type of this operation is, as you see it here, is a PDKernel. 3D, that is a positive-definite kernel. And the PDKernel class in Scalismo is the class of scalar-valued kernels. So every time now I evaluate the Gaussian kernel here on two 3-dimensional points, I get back a scalar similarity value. Now, we know that in order to build a Gaussian Process for vector-valued functions what we actually need is we need a matrix-valued kernel. And if our deformation fields are 3-dimensional, then we need a three by three actually similarity matrix between every two point values.

And in order to get this matrix-valued kernel, what you can do in Scalismo is you can use the diagonal kernel constructor here by feeding it the scalar-valued kernel. And what really happens here is that we’re actually multiplying our scalar kernel by a three by three identity matrix, a diagonal matrix. And this is how we get now a matrix-valued kernel. So now every time we evaluate a similarity using this new kernel between two 3-dimensional points, we get a three by three similarity matrix. So now that we have our two components, the mean and the covariance function, we can now build our continuous Gaussian Process.

And this we can do in Scalismo by calling the Gaussian Process constructor here and specifying the mean function and the covariance function. Let’s do this here and assign this to this gp variable. So now that we have a continuous Gaussian Process, we can now choose to evaluate it and sample deformations from it on any set of points we choose. So now I’m interested in warping a certain reference mesh that I load here and that is a scan of Paola. And then I simply sample now deformations over the points of this mesh. And this we saw how to do this before.

This is using the sample-at-points method and specifying the triangle mesh, that is also the domain, on which we want to sample, and then simply displaying the sample in the scene. When executing this line here, you might have to wait a few seconds before the result appears. So as you can see here, we now have the instance of our Gaussian Process that we just built. And we have a deformation field that is defined over this reference mesh that we just loaded.

And if we now make the mesh invisible and have a bit of a look at the deformation field, you see that we have quite a messy actually field where we have vectors that are pointing more or less in every direction, without much correlations actually, between the points of the domain, or of the mesh. Usually when we’re defining such discrete vector fields we’re interested in either deforming an image, or also a triangle mesh, using this field. So let’s have a look at what the deforming the reference with this field would give us. So I start here by actually defining a function that when given a deformation field returns me a transformation from point to point.

And this is why it actually returns me a function here that to every point associates now a new point. And what this function does is simply then finding the deformation vector corresponding to this point pt, that is actually coming from this deformation field that is passed to this function and then translating the point pt with the associated deformation vector. So if you look back at the scene this would mean if we now take, for example, this point here, we would simply move this point to the end of the tip of the vector. And once I have that function defined, what I can do is I can now apply it to the sample deformation field.

So now I get back it transformed from point to point, that I then use to warp the reference mesh using this deformation and simply then show it in the scene. And as you can see here we get a pretty messy result. And this we could have guessed actually from the discrete vector field that we saw before. So this is really the result of deforming or warping a triangle mesh with such a deformation field. And usually we’re not really interested in such results. Usually when defining free-form deformation models, we’re interested in smooth deformation that actually preserve, or do not break our meshes, as we saw in the previous case.

And to do this, or to obtain such a deformation field, we can actually first start by playing on the parameters of our Gaussian kernel. And here, for example, if you remember, we had the value l that was set to 10 in the case here. And hopefully you can work it out from this formula. You can see that if we now increase this value of l here in this formula we actually increase the correlation between distant points on our meshes.

So now we would have if, for example, we would select a value of 40 instead of 10, we would have points that are within 40 millimetres from each other that are now more correlated, according to this kernel, than if we selected a value of 10. And this is what I do now here. So I start first by cleaning up the scene. And then I just pasted together the bits of code that we executed before and simply replacing the value of l with another value that is now 40 millimetres and sampling a new deformation from so building a Gaussian Process using now this diagonalised Gaussian kernel and then sampling a deformation on the reference point, visualising it.

And also visualising its effect on the triangle mesh. So you see that we now get a much smoother deformation field with more correlation between the points, between the values or the deformation vectors, at points that are distant. So now if you look at this point here and maybe this point here, we still have although they are far away, or far apart, we still have correlated or similar deformation vectors defined. And this is why you have this smooth deformation field. And if we now look at its effect when deforming the reference mesh with this field, you see that we have a much better result than previously. So of course, this is no longer a valid face.

But this is now a smooth, deformed face. And this is a good result for our free-form deformation model. We can do more than just playing on parameters of the kernel. What we can also do and here I’m interested in symmetrising the deformations that I get out of a kernel. And what I do here is I actually apply, I turn this formula here that you see into code that when given a certain kernel k, returns me a new kernel that actually yields symmetrical deformation fields around the YZ plane that would be actually the plane in the middle of the face here.

So the way you would do that mathematically is, first of all, you would multiply your scalar-valued kernel by a diagonal matrix and then actually computing a new kernel, where before evaluating the value at the position x, you actually consider the mirror points around the YZ plane, xm here. That is a symmetrical point around the face, the middle of the face. And then you also multiply this scalar kernel by this diagonal matrix with the minus one entry here as a first entry. So if we now turn this into code, we would have the bit of code that you see here, where first actually we define our x-mirrored kernel here that is defining a new kernel for only this part here.

So this is a scalar-valued kernel. This is why it extends PDKernel. And what it actually does is when given a certain kernel ker, before it returns us actually the value of this kernel ker, only with a flipped first x-coordinate, and this is this xm coordinate that you see here. And then once we have this new scalar-valued valued kernel, we actually compute our symmetrical kernel by computing first this block here. And this is by diagonalising our kernel as we always did so far. And then to compute this part here we first take an instance of the x-mirrored kernel. So we actually create this term here.

And then we diagonalise it also using this matrix here by specifying that the first entry needs to be minus one, or multiplied by minus one, and then the kernel here is the sum of these two terms. And this is why we return the sum of these two terms, and now we defined our matrix valued kernel. So now that we have the kernel or disconstructor, what we can do is we can apply it to our Gaussian kernel, as I do here.

And then since we have our new kernel function, what we can do is we can now build a new Gaussian Process, again, with a zero mean, but now with this symmetrised kernel and then visualise, first of all, a sample deformation field from it and also its effect on the reference mesh.

And as you can see here, when we get back our sampled now deformation field, we obtain a symmetrical deformation field. And here you see the symmetry around this axis here in the middle of the face, this is the YZ plane. So in addition to the symmetry, since we actually use the Gaussian kernel to or we symmetrise the Gaussian kernel, we still have, or preserve, the properties of the Gaussian kernel. So we have still smooth deformations, deformation field that actually when we visualise now its effect on the triangle mesh, you see that we get a nice, smooth and symmetric deformation of the reference mesh. In addition to symmetrising, we can also do combinations of kernels.

So we already saw that you can add and multiply kernels. What I would like to show you here is a different type of combination of kernels that is a changepoint kernel. And here what I’m doing here is I’m defining a matrix-valued kernel, or extending the matrix-valued kernel class and defining a constructor here that takes two kernels, kernel one and kernel two. And what this kernel does is actually deactivating the kernel on one half of the face and activating the other, and vice versa on the other half. And then what I do here is I start first by actually loading a sample covariance kernel from a statistical face model.

So here I simply load the face model as we always did so far, interpolating the Gaussian Process and then taking the covariance function of this Gaussian Process. And this covariance function is actually– these are covariances that are learned from data or from the PCA model. But this is actually also a kernel that I can now combine, and I can feed it to my changepoint kernel constructor, along with, here for example, the symmetrical Gaussian kernel that I just scaled here in order to make deformations more visible.

So when I now build the Gaussian Process using this changepoint kernel and sample a deformation field from it, we will shortly see that we actually now have a sample deformation field that is split in two, where on one half of the face we have actually deformations there are coming from our sample covariance kernel, or the face-model kernel. And on the other half here, that you can see here, we have deformations that are coming from our scaled Gaussian or symmetric, symmetrised Gaussian kernel. Here the symmetry doesn’t play a role since we actually are on only one half of the face.

So I now encourage you to check the details of this last kernel and of other kernels in the companion document to this tutorial.