Hidden markov model-based korean part-of-speech tagging considering high agglutinativity, word-spacing, and lexical correlativity

Abstract

In this paper we present hidden Markov models for Korean part-of-speech tagging, which consider Korean characteristics such as high agglutinativity, word-spacing, and high lexical correlativity. In order ot consider rich information in contexts, the models adopt a less strict Markov assumption. In the models, sparse-data problem is very serious and their parameters tend to be estimated unreliably because they have a large number of parameters. To overcome sparse-data problem, our model uses a simplified version of the well-known back-off smoothing method. To mitigate unreliable estimation problem, our models assume joint independence instead of conditional independence because joint probabilities have the same degree of estimation reliability. Experimental results show that models with rich contexts perform even better than standard HMMs and that joint independent assumption is effective in some models.