International Journal of Computer Vision 71(2), 161–181, 2007
c© 2006 Springer Science + Business Media, LLC. Manufactured in the United States.
DOI: 10.1007/s11263-006-4121-7
Background-Subtraction in Thermal Imagery Using Contour Saliency
JAMES W. DAVIS AND VINAY SHARMA
Department of Computer Science and Engineering, Ohio State University, Columbus, OH 43210, USA
jwdavis@cse.ohio-state.edu
sharmav@cse.ohio-state.edu
Received April 18, 2005; Revised August 1, 2005; Accepted August 10, 2005
First online version published in June, 2006
Abstract. We present a new contour-based background-subtraction technique to extract foreground objects in
widely varying thermal imagery. Statistical background-subtraction is first used to identify local regions-of-interest.
Within each region, input and background gradient information are combined to form a Contour Saliency Map.
After thinning, an A∗ path-constrained search along watershed boundaries is used to complete and close any broken
contour segments. Lastly, the contour image is flood-filled to produce silhouettes. Results of our approach are
presented for several difficult thermal sequences and compared to alternate approaches. We quantify the results
using manually segmented thermal imagery to demonstrate the robustness of the approach.
Keywords: background subtraction, thermal imagery, infrared, FLIR, contour saliency map, CSM, video surveil-
lance and monitoring, person detection
1. Introduction
We present a new background-subtraction technique
to robustly extract foreground objects in thermal video
under different environmental conditions. Thermal
(FLIR) video cameras detect relative differences in the
amount of thermal energy emitted/reflected from ob-
jects in the scene. As long as the thermal properties
of a foreground object are slightly different (higher
or lower) from the background radiation, the corre-
sponding region in a thermal image appears at a con-
trast from the environment. Therefore thermal cameras
can be equally applicable to both day and night sce-
narios, making them a prime candidate for a persis-
tent (24-7) video system for surveillance and moni-
toring. Thermal cameras have been traditionally used
by the military for tasks such as long-range detec-
tion of enemy vehicles and Automatic Target Recog-
nition (ATR). In recent years, thermal cameras have
become increasingly employed for other applications,
including industrial inspection, surveillance, and law
enforcement.
The use of thermal imagery alleviates several classic
computer vision problems such as the presence of shad-
ows (which appear in the thermal domain only when
an object is stationary long enough for the shadow
to cool the background), lack of nighttime visibility,
and sudden illumination changes. However, thermal
imagery has its own unique challenges, including a
lower signal-to-noise ratio, uncalibrated white-black
polarity changes, and the “halo effect” that appears
around very hot or cold objects in imagery produced
by common ferroelectric BST sensors. In Fig. 1 we
show outdoor surveillance images of the same scene
captured with a thermal camera, but taken on different
days (morning and afternoon). The thermal properties
of the people and background are quite different, in-
cluding the change from bright (hot) people to dark
(cool) people in relation to the background. For such
thermal imagery to be used reliably in automatic urban

162 Davis and Sharma
Figure 1. Thermal images showing large variation in polarity and intensity.
surveillance, these image variations must be properly
addressed.
Most of the previous strategies for object/person de-
tection in thermal imagery employ “hot-spot” algo-
rithms, relying on the assumption that the object/person
is much hotter than the surrounding environment.
Though this is common in cooler nighttime environ-
ments (or during Winter), it is not always true through-
out the day or across different seasons of the year (as
convincingly shown in Fig. 1). As we will show, stan-
dard background-subtraction, image-differencing, and
hot-spot techniques are by themselves ineffective at ex-
tracting the precise locations and shapes of people in
such diverse imagery.
A prominent characteristic of thermal imagery pro-
duced from uncalibrated ferroelectric BST (chopper)
sensors is the presence of halos around objects having
a high thermal contrast with the background (Hoist,
2000). The halos have the opposite polarity (light/dark)
of the objects they surround. The strength and size of
the halos depend on factors such as the actual tem-
perature differential between the object and the sur-
rounding environment and the contrast/gain setting on
the camera. While this poses the most significant chal-
lenge to existing (and popular) background-subtraction
methodologies, we will show that our method in turn
capitalizes on this unavoidable artifact to improve per-
formance.
It should be noted here that microbolometer thermal
sensors, however, do not produce the haloing effect.
In spite of this advantage, several signal-based factors
make traditional ferroelectric BST sensors more favor-
able (Kummer, 2003). Ferroelectric BST sensors, being
AC-coupled, are better equipped to handle detector-
induced steady-state noise which can have a signifi-
cant negative impact on image quality. Also, they are
capable of resolving greater temperature variations in a
scene than the DC-coupled microbolometers. Further,
microbolometers require to recalibrate the scene at ran-
dom intervals to minimize spatial noise. This can be a
serious drawback to vision-based systems as the video
output freezes momentarily during each recalibration.
Lastly, commercially available microbolometers are
typically half the resolution of ferroelectric BST sen-
sors (with higher resolutions being considerably more
expensive). More detailed comparisons of the two sen-
sors can be found in Kummer (2003), Pandya and Anda
(2004). Due to the aforementioned issues, and that fer-
roelectric BST sensors are still in wide use by military
and law enforcement agencies, algorithms targeted for
use in the thermal domain need to be robust to the image
characteristics of ferroelectric BST sensors.
The image characteristics of thermal halos produced
by ferroelectric BST sensors can be examined using
the intensity profile of a row of pixels sliced through
an image region containing a halo. In Fig. 2(a), we
show image regions containing people, one recorded in
Summer (top), and the other recorded in Winter (bot-
tom). The images were collected from different ther-
mal cameras and at different ranges. Notice the po-
larity changes (white/black) for the people in the two
images. In Fig. 2(b), we show the corresponding back-
ground regions without the people. The row of pixels
to be examined is marked with a solid (Fig. 2(a)) and
dotted (Fig. 2(b)) line and two boundary points (left
and right) of the people have been marked with a circle.
The plots in Fig. 2(c) show the intensity profiles for the
corresponding input and background regions along the
slice. Comparison of the input intensity profiles with
those of the background makes it clear that the input
region slice is brighter/darker around the people due
to the halo, thus making the outer boundary contrast of
the people stronger.
Two key observations can be made about thermal
halos based on these plots: (1) thermal halos fade
smoothly/slowly into the image, and (2) stronger halos