Probably almost discriminative learning

Abstract

This paper develops a new computational model for learning stochasticrules, called PAD (Probably Almost Discriminative)-learning model, basedon statistical hypothesis testing theory. The model deals with theproblem of designing a discrimination algorithm to test whether or notany given test sequence of examples of pairs of (instance, label) hascome from a given stochastic rule P-{*}. Here a composite hypothesis Pis unknown other than it belongs to a given class C.In this model, we propose a new discrimination algorithm on the basis ofthe MDL (Minimum Description Length) principle, and then derive upperbounds on the least test sample size required by the algorithm toguarantee that two types of error probabilities are respectively lessthan delta(1) and delta(2) provided that the distance between the tworules to be discriminated is not less than epsilon.For the parametric case where C is a parametric class, this paper showsthat an upper bound on test sample size is given by O(1/epsilon ln1/delta 1 + 1/(epsilon)2 ln 1/delta(2) + fake + k/epsilon ln k/epsilon +l(M)/epsilon). Here k is the number of real-valued parameters for thecomposite hypothesis P, and l(M) is the description length for thecountable model for P. Further this paper shows that the MDL-baseddiscrimination algorithm performs well in the sense of sample complexityefficiency, comparing it with other kinds of information-criteria-baseddiscrimination algorithms. This paper also shows how to transform anystochastic PAC (Probably Approximately Correct)-learning algorithm intoa PAD-learning algorithm.For the non-parametric case where C is a non-parametric class but thediscrimination algorithm uses a parametric class, this paperdemonstrates that the sample complexity bound for the MDL-baseddiscrimination algorithm is essentially related to Barren and Cover'sindex of resolvability. The sample complexity bound gives a new view atthe relationship between the index of resolvability and the MDLprinciple from the PAD-learning viewpoint.