Median filtering and background images

Median filtering over the image space

Filtering is a process by which we use the values of pixels to calculate a new value in their place. By calculating an average over a window for pixels, for example, we can blur the image spatially.

For example, we can calculate a mean of the values in the orange pixels below, and replace the red pixel value with this calculated average. If we move this window across the image and repeat this calculation, we blur the image.

Instead of calculating a mean, we could calculate a different average such as a median. A median average first orders the data by size, then picks the middle value. The advantage of a median is it is not skewed by a small number of extra large values; see the figure below.

The median of the numbers above is 7; the mean is 20.1 . You can see in this case how the mean is more affected by the extreme value (96) than the median. Also note that instead of 96 this value could be exceedingly large (say, ten million!), and the median would not change.

Median filtering over time

The same kind of filtering can be used over time, instead of space, to give a median pixel value over a time window. The seven numbers in the previous figure could be thought of as values of a single pixel, changing over time in a video sequence. The lower numbers in this example (e.g. 4-12) might be the value of the background at that pixel – perhaps a brick wall – varying slightly due to image noise. Perhaps something in the foreground – a leaf blowing in the wind – moves in front of the brick pixel for a single frame – this is the 96 value.

As you can see, median filtering will completely ignore the single bright pixel, and instead show one of the background values (7 in this case). By following how median filtering works, we can work out that we are allowed three frames of foreground object in the window of seven frames above before a foreground value will actually be shown. This is demonstrated below, where 4 of our 7 frames now has values deriving from a foreground leaf:

So, if we have a time window of N frames, as long as more than N/2 are background, a background image will be shown. This means we can calculate what the background of a scene looks like, even when foreground objects are moving around the scene, as long as they do not dwell over a particular point for more than half the time window. This is how median filtering can be used to produce a background image.