On asymptotically optimal methods of prediction and adaptive coding for Markov sources

Abstract

The problem of predicting a sequence x1, x2,... generated by a discrete source with unknown statistics is considered. Each letter xt+1 is predicted using information on the word x1x2...x1 only. In fact, this problem is a classical problem which has received much attention. Its history can be traced back to Laplace. To estimate the efficiency of a method of prediction, three quantities are considered: the precision as given by the Kullback-Leibler divergence, the memory size of the program needed to implement the method on a computer, and the time required, measured by the number of binary operations needed at each time instant. A method is presented for which the memory size and the average time are close to the minimum. The results can readily be translated to results about adaptive coding. © 2001 Elsevier Science (USA).