New statistical methods for phrase break prediction

Abstract

The paper presents two methods for the prediction of phrase breaks. The first method uses a standard HMM part-of-speech tagger with variable context length. The second method directly encodes the distance from the last phrase break in its states. It combines the probability of a phrase break given the distance from the last phrase break with the probability of a break given the local context consisting of the surrounding words and part of speech tags. The accuracy of the new tagger is 2 percentage points higher than that of Taylor and Black (1998) on similar data.