Model-based Image Analysis : Active Contours or Snakes

A look at the model-based image analysis approach, and a more detailed description of one such approach, active contours or snakes.

Model-based image analysis

Model-based image analysis is distinguished from other approaches by its use of an explicit model of the object, or more commonly class of objects, being analysed. Of course, all image analysis programs incorporate some knowledge of the task at hand, but in most it is implicit – embedded in the sequence of processes applied and their parameters. In model-based systems the knowledge being used can be seen outside of the code, perhaps as an equation, perhaps as a data structure.

A tool that detects humans in images might, for example, rely on a stick-figure representation of an idealised human body that is overlaid on the image as analysis proceeds. As key features in the model – maybe skin-coloured regions at the ends of the arms and neck – are matched to image features the model is parameterised.

When analysis begins the model captures the possible appearances of a human, but the specific features of the person shown in the image – e.g. the lengths of and current angles between their limbs – are unknown. When it is complete, the fully parameterised model reflects the image content, and describes an individual human.

Active contours / Snakes

In the model-based approach the model also directs analysis by restricting its parameters to sets of allowable values: e.g. identifying the position of a hand limits the number of image locations which might reasonably depict an elbow. The model used might be two -or three-dimensional, in which case the position and orientation of the model in space might be determined as well at its internal parameters.

Though there is a growing literature on the construction of 3D descriptions of plants, both from sensor data and via functional-structural modelling, their physical complexity means that 3D model-based approaches to plant image analysis are rare. Two-dimensional methods are more feasible at present, with the use of active contours, or “snakes”, perhaps the most common technology.

Active contours model the boundaries of objects by imposing constraints on the relative positions of points on those boundaries. They are often referred to as snakes because an early paper use the analogy of a snake trying to make itself comfortable while lying on a surface. Though the snake can bend to fit the ridges and troughs in the surface to some degree, its physicality means that not all configurations are possible. It cannot perform e.g. a 90 degree bend. Central to the active contour approach is the notion of an energy function. This scores the current configuration of a snake on a number of criteria, assigning it an energy. During image analysis the points on the contour are moved in an effort to minimise that energy and fit the snake to a boundary in the input image.

Energy functions comprise two major components, the internal and external energies. The internal energy term captures the physical properties of the snake. A common way of computing internal energy is to estimate curvature at each point and sum these estimates to provide a measure of the smoothness of the contour. Smoother (low curvature) contours are typically assigned lower energies. The external energy assesses the fit of the contour to the image. It might, for example, estimate the image gradient under each point and give low energies to snakes which lie on points with high gradient values.

The total energy is a weighted sum of these two terms:

[E_{total} = alpha E_{internal} + beta E_{external}]

[E_{total} = alpha E_{internal} + beta E_{external} + chi E_{constraint}]

This assigns low energy to contours with either closely or widely-spaced points and so, when other energies are low, causes the contour to either shrink or expand across the image. The figure below shows the result of shrinking a snake seeking a smooth, high image gradient contour onto a leaf. The blue line is the starting position, red lines show the contour position after every 100 hill climbing iterations, and the green the final contour achieved after 750 iterations.

Other kinds of models

We have looked at shape models here, but it is common to use other models to help segment image data. For example, we can have a model of the image features we expect an object to be composed of – things like the expected colour, texture, etc. of an object. This kind of model doesn’t match the shape, but rather the look of an object.

In the following practical we will walk you through one of these approaches, which learns to find the features like colour and texture which describe an object using machine learning, and then uses this model to mark object regions in the image. You will notice that without a model of shape (versus with Snakes) the boundaries of segmented areas with such approaches can be more noisy, and you can get different regions of segmented areas throughout the image, rather than in one place.

So sometimes it may be more appropriate to use a shape model, sometimes a feature-based model, sometimes a combination of both.