Ptr4BERT: Automatic Semisupervised Chinese Government Message Text Classification Method Based on Transformer-Based Pointer Generator Network

Abstract

With the development of Internet technology, government affairs can be handled online. More and more citizens are using online platforms to report to government departments, which is generating a lot of textual data. Among them, the basic but important problem is to automatically classify the different categories of messages, so that staff from different departments can process relevant information quickly. However, government messages have problems such as fast update rate, a large amount of information, long texts, and difficulty in capturing key points, which make supervised learning methods unsuitable for processing such texts. To address these problems, we propose a semisupervised text classification method based on a transformer-based pointer generator network named Ptr4BERT, which uses the pointer generator network with BERT(bidirectional encoder representation from transformers) embedding as a preprocessor for feature extraction. In this method, text classification can achieve very good results with a small set of labeled data, by extracting features exclusively from the message text. In order to verify the effect of our proposed model, we performed some experiments. Besides, we designed a crawler program and obtained two datasets from different websites, which are named HNMes and QDMes. Experimental results have shown that the proposed method outperforms the state-of-the-art methods significantly.