Unsupervised word segmentation for sesotho using adaptor grammars

Abstract

This paper describes a variety of nonparametric Bayesian models of word segmentation based on AdaptorGrammars that model different aspects of the input and incorporate different kinds of prior knowledge, and applies them to the Bantu language Sesotho. While we find overall word segmentation accuracies lower than these models achieve on English, we also find some interesting differences in which factors contribute to better word segmentation. Specifically, we found little improvement to word segmentation accuracy when we modeled contextual dependencies, while modeling morphological structure did improve segmentation accuracy.