AUG-BERT: An Efficient Data Augmentation Algorithm for Text Classification

Abstract

We propose a BERT-based data augmentation for labeled sentences called Aug-Bert. New sentences are generated by stochastically selecting words and replacing them with other words predicted by Aug-BERT. After a two-stage training, including CLP and L-MLM, BERT can be fine-tuned to be Aug-BERT. Aug-BERT can predict stochastically selected words according to both the context and the label. Depending on the ability of BERT’s deep bidirectional language model and the information of label incorporated via label-segment embedding, Aug-BERT can generate sentences of high quality. Experiments on six different text classification tasks show that our methods outperform most of the others and can improve the performance of classifiers obviously.