Unsupervised Traditional Chinese Medicine Text Segmentation Combined with Domain Dictionary

Abstract

The literature in the field of traditional Chinese medicine (TCM) contains a large amount of knowledge of traditional Chinese medicine. Such knowledge plays an important role in the automatic diagnosis and treatment of TCM. In order to obtain the above knowledge, the word segmentation of TCM texts is crucial and fundamental. However, the corpus of TCM texts presents mostly in the form of classical Chinese, which is different from modern Chinese, but there are still some cases where the path of modern Chinese and classical Chinese comes in cross. So it is very difficult to use the current universal word segmentation. The lack of manually labeled corpus for TCM word segmentation which is also hard to get makes it difficult to use a supervised method to train the tokenizer applied in the Chinese medicine. In order to solve this problem, we use an unsupervised method which uses entropy as goodness together with traditional Chinese medicine domain dictionary to construct a TCM text-specific tokenizer. Finally, the effectiveness of this method has been proved through experiments on 280 MB TCM texts.