A Case Study of Multi-class Classification with Diversified Precision Recall Requirements for Query Disambiguation

Abstract

We introduce a new metric for measuring the performance of multi-class classifiers. This metric is a generalization of the f1 score that is defined on binary classifiers, and offers significant improvement over other generalizations such as micro-and macro-averaging. In particular, one can select coefficients that weight the per-class precision and recall, as well as the overall class importance, with a robust mathematical interpretation. When certain parameters are selected our metric yields macro-averaged statistic as a special case. We demonstrate the efficacy of this metric on an application in genealogical search.