A new method for weighted ensemble clustering and coupled ensemble selection

Abstract

Clustering ensemble, also referred to as consensus clustering, has emerged as a method of combining an ensemble of different clusterings to derive a final clustering that is of better quality and robust than any single clustering in the ensemble. Normally clustering ensemble algorithms in the literature combine all the clusterings without learning the ensemble. But by learning the ensemble, one can define the merit of a clustering or even a cluster in it, in forming a quality consensus. In this work, we propose a cluster-level surprisal measure to define the merit of a clustering that reflects both levels of agreement as well as disagreement among clusters. Using the proposed measure of merit, we devise a polynomial heuristics that judiciously selects a subset of clusterings from the ensemble that contribute positively in forming the consensus. We also empirically show that consensus achieved by our proposed method performs better in terms of quality compared to well-known clustering ensemble algorithms on different benchmark datasets.