Learning to shape errors with a confusion objective

Abstract

Errors are the enemy of classification systems, so minimising the total probability of error is an understandable objective in statistical machine learning classifiers. However, for open-world application in trusted autonomous systems, not all errors are equal in terms of their consequences. So, the ability for users and designers to define an objective function that distributes errors according to preference criteria might elevate trust. Previous approaches in cost-sensitive classification have focussed on dealing with distribution imbalances by cost weighting the probability of classification. A novel alternative is proposed that learns a ‘confusion objective’ and is suitable for integration with modular Deep Network architectures. The approach demonstrates an ability to control the error distribution in training of supervised networks via back-propagation for the penalty of an increase in total errors. Theory is developed for the new confusion objective function and compared with cross-entropy and squared loss objectives. The capacity for error shaping is demonstrated via a range of empirical experiments using a shallow and deep network. The classification of handwritten digits from up to three independent databases demonstrates desired error performance is maintained across unforeseen data distributions. Some significant and unique forms of error control are demonstrated and their limitations investigated.