Employ decision values for soft-classifier evaluation with crispy references

Abstract

Evaluation of classification performance has been comprehensively studied for both crispy and fuzzy classification tasks. In this paper, we address the hybrid case: evaluating fuzzy prediction results against crispy references. The proposal is motivated by the following facts: (1) most datasets in practice are produced with crispy labels due to the excessive cost of fuzzy labelling; and (2) many state-of-the-art classifiers can yield fuzzy decision values even if they are trained from data with crispy labels. We derive our fuzzy-crispy evaluation criterion based on a widely adopted fuzzy-set-based evaluation method. By exploiting the distribution of decision values, the proposed criterion bears more comprehensive information than conventional crispy classification evaluation criteria. The advantages of the proposed criterion are demonstrated in artificial and real-world classification case studies.