Fast Condensed Nearest Neighbor Rule
Fabrizio Angiulli angiulli@icar.cnr.it
ICAR-CNR, Via Pietro Bucci 41C, 87036 Rende (CS), Italy
Abstract
We present a novel algorithm for computing a
training set consistent subset for the nearest
neighbor decision rule. The algorithm, called
FCNN rule, has some desirable properties.
Indeed, it is order independent and has sub-
quadratic worst case time complexity, while
it requires few iterations to converge, and it
is likely to select points very close to the de-
cision boundary. We compare the FCNN rule
with state of the art competence preservation
algorithms on large multidimensional train-
ing sets, showing that it outperforms existing
methods in terms of learning speed and learn-
ing scaling behavior, and in terms of size of
the model, while it guarantees a comparable
prediction accuracy.
1. Introduction
The nearest neighbor decision rule (Cover & Hart,
1967) (NN rule in the following) assigns to an un-
classified sample point the classification of the near-
est of a set of previously classified points. For this
decision rule, no explicit knowledge of the underlying
distributions of the data is needed. A strong point of
the nearest neighbor rule is that, for all distributions,
its probability of error is bounded above by twice the
Bayes probability of error (Cover & Hart, 1967; Stone,
1977; Devroye, 1981). That is, it may be said that half
the classification information in an infinite size sample
set is contained in the nearest neighbor. Naive im-
plementation of the NN rule requires to store all the
previously classified data, and to compare then each
sample point to be classified to each stored point. In
order to reduced both space and time requirements,
several techniques to reduce the size of the stored data
for the NN rule have been proposed (see (Wilson &
Martinez, 2000; Toussaint, 2002) for a survey) referred
Appearing in Proceedings of the 22nd International Confer-
ence on Machine Learning, Bonn, Germany, 2005. Copy-
right 2005 by the author(s)/owner(s).
to as training set reduction, training set condensation,
reference set thinning, and prototype selection algo-
rithms. In particular, among these techniques, train-
ing set consistent ones, aim at selecting a subset of
the training set that classifies the remaining data cor-
rectly through the NN rule. Using a training set con-
sistent subset, instead of the entire training set, to
implement the NN rule, has the additional advantage
that it may guarantee better classification accuracy.
Indeed, (Karac¸ali & Krim, 2002) showed that the VC
dimension of a NN classifier is given by the number
of reference points in the training set. Thus, in order
to achieve a classification rule with controlled gener-
alization, it is better to replace the training set with
a small consistent subset. Unfortunately, computing
a minimum cardinality training set consistent subset
for the NN rule turns out to be intractable (Wilfong,
1992). Several training set consistent condensation al-
gorithms have been introduced in literature. We point
out that, among the criteria characterizing condensa-
tion methods, the learning speed one is usually ne-
glected. But, in order to manage huge amounts of
data, methods exhibiting good scaling behavior are
definitively needed. In this work we present a novel
order independent algorithm for finding a training set
consistent subset for the NN rule, called FCNN rule,
and compare it with existing methods. The rest of
the paper is organized as follows. In Section 2, exist-
ing approaches are briefly described and compared to
the approach here proposed. In Section 3, the FCNN
rule is described and its main properties are stated. In
Section 4, experimental results are presented together
with a thorough comparison with existing methods.
Finally, in Section 5, conclusions are drawn.
2. Related Works and Contribution
Starting from the seminal work of (Hart, 1968), several
training set condensation algorithms have been intro-
duced, also known as instance-based, lazy, memory-
based, and case-based learners. These methods can be
grouped into three categories depending on the objec-
tives that they want to achieve (Brighton & Mellish,