A Bayesian Interpretation of Interpolated Kneser-Ney

Abstract

Interpolated Kneser-Ney is one of the best smoothing methods for n-gram language models. PRevious explanations for its superiority have been based on intuitive and empirical justifications of specific properties of the method. We preopose a novel interpretation of interpolated Kneser-Ney as approximate inference in a hierarchical Bayesian model consisting of Pitman-Yor processes. As opposed to past explanations, our interpretation can recover exactly the formulation of interpolated Kneser-Ney, and performs better than interpolated Kneser-Ney when a better inference procedure is used.