Cluster validation for high-dimensional datasets

Abstract

Cluster validation is the process of evaluating performance of clustering algorithms under varying input conditions. This paper presents a new solution to the problem of cluster validation in high-dimensional applications. We examine the applicability of conventional cluster validity indices in evaluating the results of high-dimensional clustering and propose new indices that can be applied to high-dimensional datasets. We also propose an algorithm for automatically determining cluster dimension. By utilizing the proposed indices and the algorithm, we can discard the input parameters that PROCLUS needs. Experimental studies show that the proposed cluster validity indices yield better cluster validation performance than is possible with conventional indices. © Springer-Verlag Berlin Heidelberg 2004.