Feature selection with rényi min-entropy

Abstract

We consider the problem of feature selection, and we propose a new information-theoretic algorithm for ordering the features according to their relevance for classification. The novelty of our proposal consists in adopting Rényi min-entropy instead of the commonly used Shannon entropy. In particular, we adopt a notion of conditional min-entropy that has been recently proposed in the field of security and privacy, and which is strictly related to the Bayes error. We evaluate our method on two classifiers and three datasets, and we show that it compares favorably with the corresponding one based on Shannon entropy.