We present a simple approach for Asian language text classification without word segmentation, based on statistical-gram language modeling. In particular, we examine Chinese and Japanese text classification. With character-gram models, our approach avoids word segmentation. However, unlike traditional ad hoc-gram models, the statistical language modeling based approach has strong information theoretic basis and avoids explicit feature selection procedure which potentially loses significantly amount of useful information. We systematically study the key factors in language modeling and their influence on classification. Experiments on Chinese TREC and Japanese NTCIR topic detection show that the simple approach can achieve better performance compared to traditional approaches while avoiding word segmentation, which demonstrates its superiority in Asian language text classification.
CITATION STYLE
Peng, F., Huang, X., Schuurmans, D., & Wang, S. (2003). Text classification in Asian languages without word segmentation. In Proceedings of the 6th International Workshop on Information Retrieval with Asian Languages, IRAL 2003 (pp. 41–48). Association for Computational Linguistics (ACL). https://doi.org/10.3115/1118935.1118941
Mendeley helps you to discover research relevant for your work.