CANCN-BERT: A Joint Pre-Trained Language Model for Classical and Modern Chinese

Abstract

Pre-Trained Models (PTMs) can learn general knowledge representations and perform well in Natural Language Processing (NLP) tasks. For the Chinese language, several PTMs are developed, however, most existing methods concentrate on modern Chinese and are not ideal for processing classical Chinese due to the differences in grammars and semantics between these two forms. In this paper, in order to process two forms of Chinese uniformly, we propose a novel Classical and Modern Chinese pre-trained language model (CANCN-BERT), with the advantage of effectively processing both classical and modern Chinese, which is an extension of BERT. Form-aware pre-training tasks are elaborately designed to train our model, so as to better adapt it to classical and modern Chinese corpus. Moreover, we define a joint model, proposing dedicated optimization methods through different paths with the control of the switch mechanism. Our model merges characteristics of both classical and modern Chinese, which can adequately and efficiently enhance the representation ability for both forms. Extensive experiments show that our model outperforms baseline models on processing classical and modern Chinese and achieves significant and consistent improvements. Also, the results of ablation experiments demonstrate the effectiveness of each module.