MIHNet: Combining N-gram, Sequential and Global Information for Text Classification

Abstract

Pre-Training language model has achieved amazing results in many NLP tasks. In Particular, BERT (Bidirectional Encoder Representations from Transformers) create a new era in NLP tasks. Despite the success, these model perform well at Global Information but weak on n-gram and Sequential information. In this paper, we conduct exhaustive experiments of classical text classification models upon BERT in text classification task and provide a general guide for BERT+ models. Finally, we propose a new text classification model called MIHNet (Multi-dimension Information Integration using Highway network), which integrates Global, n-gram and Sequential information together and get a better performance. Notably, our model obtains new state-of-The-Art results on eight widely-studied text classification datasets.