Lower Bounds on Learning Decision Lists and Trees

Abstract

k-Decision lists and decision trees play important roles in learning theory as well as in practical learning systems. k-Decision lists generalize classes such as monomials, k-DNF, and k-CNF, and like these subclasses they are polynomially PAC-learnable [R. Rivest, Mach. Learning 2 (1987), 229-246]. This leaves open the question of whether k-decision lists can be learned as efficiently as k-DNF. We answer this question negatively in a certain sense, thus disproving a claim in a popular textbook [M. Anthony and N. Biggs, "Computational Learning Theory," Cambridge Univ. Press, Cambridge, UK, 1992]. Decision trees, on the other hand, are not even known to be polynomially PAC-learnable, despite their widespread practical application. We will show that decision trees are not likely to be efficiently PAC-learnable. We summarize our specific results. The following problems cannot be approximated in polynomial time within a factor of 2logδ n for any δ < 1, unless NP ⊂ DTIME[2polylog n]: a generalized set cover, k-decision lists, k-decision lists by monotone decision lists, and decision trees. Decision lists cannot be approximated in polynomial time within a factor of nδ, for some constant δ > 0, unless NP=P. Also, k-decision lists with / 0-1 alternations cannot be approximated within a factor logl n unless NP ⊂ DTIME[no(log log n)] (providing an interesting comparison to the upper bound obtained by A. Dhagat and L. Hellerstein [in "FOCS '94," pp. 64-74]). © 1996 Academic Press, Inc.