Accelerating Semi-Supervised Text Classification by K-Way Projecting Networks

Abstract

The state of the art semi-supervised learning framework has greatly shown its potential in making deep and complex language models such as BERT highly effective for text classification tasks when labeled data is limited. However, the large size and low inference speed of such models may hinder their application on resources-limited or real-time use cases. In this paper, we propose a new approach in semi-supervised learning framework to distill large complex teacher model into a fairly lightweight student model which has the ability of acquiring knowledge from different layers of teacher with the usage of K-way projecting networks. Across four English datasets in text classification benchmarks and one dataset collected from an Chinese online course, our experiment shows that this student model achieves comparable results with the state of the art Transformer-based semi-supervised text classification methods, while using only 0.156MB parameters and having an inference speed 785 times faster than the teacher model.