Characterizing Rational versus Exponential Learning Curves

Abstract

We consider the standard problem of learning a concept from random examples. Here a learning curve is defined to be the expected error of a learner's hypotheses as a function of training sample size. Haussler, Littlestone, and Warmuth have shown that, in the distribution-free setting, the smallest expected error a learner can achieve in the worst case over a class of concepts C converges rationally to zero error; i.e., Θ(t-1) in the training sample size t. However, Cohn and Tesauro have recently demonstrated that exponential convergence can often be observed in experimental settings (i.e., average error decreasing as eΘ( -t)). By addressing a simple non-uniformity in the original analysis this paper shows how the dichotomy between rational and exponential worst case learning curves can be recovered in the distribution-free theory. In particular, our results support the experimental findings of Cohn and Tesauro: for finite concept classes any consistent learner achieves exponential convergence, even in the worst case, whereas for continuous concept classes no learner can exhibit sub-rational convergence for every target concept and domain distribution. We also draw a precise boundary between rational and exponential convergence for simple concept chains - showing that somewhere-dense chains always force rational convergence in the worst case, while exponential convergence can always be achieved for nowhere-dense chains. © 1997 Academic Press.