Training on the Test Set: Mapping the System-Problem Space in AI

Abstract

Many present and future problems associated with artificial intelligence are not due to its limitations, but to our poor assessment of its behaviour. Current evaluation practices produce aggregated performance metrics that lack detail and quantified uncertainty about the following question: how will an AI system, with a particular profile π, behave for a new problem, characterised by a particular situation µ? Instead of just aggregating test results, we can use machine learning methods to fully capitalise on this evaluation information. In this paper, we introduce the concept of an assessor model, (equation presented)(r|π, µ), a conditional probability estimator trained on test data. We discuss how these assessors can be built by using information of the full system-problem space and illustrate a broad range of applications that derive from varied inferences and aggregations from (equation presented). Building good assessor models will change the predictive and explanatory power of AI evaluation and will lead to new research directions for building and using them. We propose accompanying every deployed AI system with its own assessor.