Skip to 0 minutes and 7 seconds Welcome to week two of the shape modelling course. In this video, we’re going to discuss how we can model shape deformations using the concept of a Gaussian Process.

Skip to 0 minutes and 20 seconds This is maybe the most important theory video of the course as we will introduce the basic concept on which everything that we will discuss in this course is based. In case you should feel a bit lost at some point, please don’t worry. The course is designed such that the concept becomes more clear as we go along and especially also when you do the exercises. Last week we have measured the length and span of our hands. For this we defined four points and measured the distance between these points.

Skip to 1 minute and 0 seconds Assume we are given a new shape together with a vector that tells us where the points are located in the new shape. It would then be simple to repeat the exact same measurement for the new shape and compare the lengths and span of the two hands. This measurement is interesting and already tells us something about the shape differences of the two hands.

Skip to 1 minute and 28 seconds If you knew from this deformation vector not only four points, but many more points, we would have much more information about how the two shapes differ. Another way of thinking about it is that the original hand, together with this deformation vectors, actually define the hand shape. This is exactly the point of view that we take in this course. Instead of modelling different shapes directly, we model shape changes as deformations from a given shape. Or in other words, how do points move between the shape. Starting from the same reference shape here, we will now define two different shapes by inducing two different types of deformation fields. Let us define this idea more formally.

Skip to 2 minutes and 21 seconds We fix an arbitrary shape and call it the reference shape. We denote it by gamma R. Gamma R is just a set of points. It can finite or it can be infinite. To describe a shape variation, we define a vector field u which maps from the reference shape to R2. For a fixed reference shape, we can reformulate the problem of shape modelling as the problem of modelling shape deformations. Or, alternatively, how the points can move. Let’s see how we can model this shape deformation. We start by specifying the deformation u of x at one point x. Note that we need to model two components, the deformation in x and the deformation in y direction.

Skip to 3 minutes and 20 seconds We assume that the deformation follows a multivariate normal distribution. To specify the model we need to define the mean deformation. This is the deformation that will move the point to the position where the tip of the thumb lies in an average shape. We also need to specify the covariance matrix. The covariance matrix defines how much the position of the tip of the thumb can vary. This is specified by this 2 times 2 matrix. The first component defines the variance in the x direction, while the second component– here, sigma 2-2 – defines the variance in y direction. In addition, we need to model how the deformation in x and the deformation in y direction correlate.

Skip to 4 minutes and 13 seconds Which is done by this entry sigma 2-1 and this one, sigma 1-2.

Skip to 4 minutes and 21 seconds The model becomes more interesting if we include a second point. As you want to model the correlations between the two deformations, we model their joint distribution using, again, a multivariate normal distribution. The mean and covariance consist now of several blocks.

Skip to 4 minutes and 47 seconds We have the mean deformation for the point x, the covariance for the point x, and the same for the point x prime. In addition, we now also need to model the two correlations between x and x prime which is done by specifying the covariance between each individual component x and y of the two points. This model can be generalised to any number of points. However, it is a little bit cumbersome to specify all the values in a matrix for such a large number of points. We therefore assume that we have a mean function available which assigns to every point of the reference here a vector in R2. So where this point would move on average.

Skip to 5 minutes and 39 seconds Likewise, we assume that we have to covariance function which we call K. And this covariance function defines the covariance between the two points, which in this case is a 2 times 2 matrix. Using these two functions we can now compute for any finite set the corresponding normal distribution. This is done by simply evaluating the two functions of every point. So for the mean function, we evaluate this function for all the points of our reference. We do the same for the covariance function where we evaluate it at all the pairs of the given reference.

Skip to 6 minutes and 26 seconds This results in a huge joint normal distribution. However, we still recognise that the mean and the covariance matrix have the same structure as before.

Skip to 6 minutes and 41 seconds The model that we have just defined is called a Gaussian Process model. It is a model that simply specifies a multivariate normal distribution by giving a mean and a co-variant function. It is a bit more general than a standard multivariate normal distribution. As it turns out, that it also works if we defined a mean and covariance function on an infinite number of points. To summarise, a Gaussian Process is an extension of the multivariate normal distribution. While we think of a multivariate normal distribution as giving a distribution over vectors, we think of a Gaussian Process as providing a distribution over functions. Which are, in our case, vector fields that model shape deformation.

Skip to 7 minutes and 30 seconds Gaussian Processes have been used for decades in statistics and are also a very popular tool in machine learning. This has the huge advantage that there are many well established and useful concepts that we can just use for shape modelling. We will explore some of them later in this course.

# Modelling shape deformations

The central question in shape modelling is how to model the shape variations within a shape family.

In this course, the answer to this question is by means of the normal distribution. In this video we will explain what this means, how we can represent shape changes as deformations, and how the concept of a *Gaussian Process* can be used to formulate our model assumptions.

© University of Basel