s
v
*,
ersit
005
e 25
Da
Abstract
relative differences in the amount of thermal energy
are therefore independent of illumination, making them
is beset by the presence of shadows, sudden illumination
changes, and poor nighttime visibility, thermal imagery
has its own unique challenges. The commonly used ferro-
electric BST (chopper) thermal sensor yields imagery with
a low signal-to-noise ratio, uncalibrated white-black
polarity changes, and the ‘‘halo effect’’ that appears around
* Corresponding author. Fax: +1 614 292 2911.
E-mail addresses: jwdavis@cse.ohio-state.edu (J.W. Davis), sharmav@
cse.ohio-state.edu (V. Sharma).
Computer Vision and Image UnderstaOne of the most desirable qualities of a video surveil-
lance system is persistence, or the ability to be effective at
all times (day and night). However a single sensor is
generally not effective in all situations (e.g., a color camera
at night). To attain persistence, we present a new back-
ground-subtraction technique to segment foreground
objects that relies on the integration of two complementary
bands of the electromagnetic spectrum, long-wave infrared
(thermal) and visible light.
Thermal (FLIR) and color video cameras are both
widely used for surveillance. Thermal cameras detect
more effective than color cameras under poor lighting con-
ditions. Color optical sensors on the other hand are obliv-
ious to temperature differences in the scene, and are
typically more effective than thermal cameras when objects
are at ‘‘thermal crossover’’ (thermal properties of the object
are similar to the surrounding environment), provided that
the scene is well illuminated and the objects have color sig-
natures different from the background.
In order to exploit the enhanced potential of using both
sensors together, one needs to address the computer vision
challenges that arise in both domains. While color imageryWe present a new background-subtraction technique fusing contours from thermal and visible imagery for persistent object detection
in urban settings. Statistical background-subtraction in the thermal domain is used to identify the initial regions-of-interest. Color and
intensity information are used within these areas to obtain the corresponding regions-of-interest in the visible domain. Within each
region, input and background gradient information are combined to form a Contour Saliency Map. The binary contour fragments,
obtained from corresponding Contour Saliency Maps, are then fused into a single image. An A* path-constrained search along watershed
boundaries of the regions-of-interest is used to complete and close any broken segments in the fused contour image. Lastly, the contour
image is flood-filled to produce silhouettes. Results of our approach are evaluated quantitatively and compared with other low- and high-
level fusion techniques using manually segmented data.
 2006 Elsevier Inc. All rights reserved.
Keywords: Background-subtraction; Fusion; Thermal imagery; Infrared; FLIR; Contour Saliency Map; CSM; Video surveillance and monitoring; Person
detection
1. Introduction emitted/reflected from objects in the scene. These sensorsBackground-subtraction u
of thermal and
James W. Davis
Department of Computer Science and Engineering, Ohio State Univ
Received 23 November 2
Available onlin
Communicated by James1077-3142/$ - see front matter  2006 Elsevier Inc. All rights reserved.
doi:10.1016/j.cviu.2006.06.010ing contour-based fusion
isible imagery
Vinay Sharma
y, 491 Dreese Lab, 2015 Neil Avenue, Columbus, OH 43210, USA
; accepted 15 June 2006
January 2007
vis and Riad Hammoud
www.elsevier.com/locate/cviu
nding 106 (2007) 162–182

and across, the two domains. The approach is also well-
suited to handle the typical problems in both domains
(e.g., shadows, thermal halos, and polarity changes). The
method does not rely on any prior shape models or motion
information, and therefore could be particularly useful for
bootstrapping more sophisticated tracking techniques. The
method is based on our previous approach [11,10] for
object detection in thermal imagery.
In Fig. 2, we show a flowchart of the proposed algorithm.
We start by identifying preliminary regions-of-interest
(ROIs) in the two domains via standard Background-
Subtraction. In this stage, the ROIs obtained in the thermal
domain are used to localize the background-subtraction
operation in the visible domain, shown by the dotted arrow
J.W. Davis, V. Sharma / Computer Vision and Image Understanding 106 (2007) 162–182 163very hot or cold objects (see Fig. 1, showing a bright halo
around dark (colder) person regions in a hot environment).
The halo effect is caused due to the AC-coupling in ferro-
electric focal plane arrays, that results in a droop/under-
shoot [29] in the response to uniformly hot and cold
objects in the scene. Gain and level settings, typically used
to obtain high contrast imagery with sharp object bound-
aries, further enhance this haloing effect and make auto-
matic shape segmentation from thermal imagery very
difficult.
These challenges of thermal imagery have been largely
ignored in the past by algorithms (‘‘hot spot’’ techniques)
based on the highly limiting assumption that the target
object (aircraft, tank, person) is much hotter than the
surrounding environment. For surveillance and other
applications involving the monitoring of people, this
assumption is valid only in certain conditions like cooler
nighttime environments (or during Winter); it is not
always true throughout the day or for different seasons
of the year.
We propose an enhanced background-subtraction algo-
Fig. 1. Thermal image showing bright halo around dark person regions.rithm using both visible and thermal imagery. The
approach makes use of region- and gradient-based process-
ing to highlight contours that are the most salient within,
Fuse
Background-
Subtraction
Background-
Subtraction
Contour
Extraction
Contour
Extraction
Thermal
Imagery
Visible
Imagery
I. Low-Level II. Mid-Le
Fig. 2. Flowchart of propin the flowchart. Next, in the Contour Extraction stage, we
identify salient contour segments corresponding to the fore-
ground object(s) within the ROIs of both domains by utiliz-
ing the input and background gradient information. We
then Fuse the contours from corresponding ROIs using
the combined input gradient information from both
domains. In the Silhouette Creation stage we first close
and complete the contours using an A* search algorithm
constrained to a local watershed segmentation and then
flood-fill the contours to create silhouettes. In the final
Post-Processing stage we eliminate regions based on a min-
imum size threshold, and also use temporal filtering to
remove sporadic detections. We then assign to each remain-
ing silhouette a confidence value representative of how dif-
ferent it is from the background.
As shown in the figure, the entire pipe-line can be
divided into three main processing levels. The low-level
stage (Stage I) of processing deals directly with raw pixel
intensities, the mid-level stage (Stage II) involves the extrac-
tion and manipulation of features, and the high-level stage
(Stage III) refines the results and operates on decisions
made by the lower levels. The level at which a fusion algo-
rithm combines information from the different input sen-
sors can play a vital role in object-detection performance.
Our algorithm is a mid-level fusion technique as it fuses
information at the contour level. The contour features
extracted allow the algorithm to focus on high intensity
Post-ProcessingSilhouette Creation
Final
Silhouettes
vel III. High-Levelosed fusion algorithm.