PAC learning using nadaraya-watson estimator based on orthonormal systems

Abstract

Regression or function classes of Euclidean type with compact support and certain smoothness properties are shown to be PAC learnable by the Na~laraya-Watson estimator based on complete orthonormal systems. While requiring more smoothness properties than typical PAC formulations, this estimator is computationally efficient, easy to implement, and known to perform well in a number of practical applications. The sample sizes necessary for PAC learning of regressions or functions under sup norm cost are derived for a general orthonormal system. The result covers the widely used estimators based on Haar wavelets, trignometric functions, and Daubechies wavelets.