Feature Extraction by Using Attention Mechanism in Text Classification

Abstract

In recent years, machine learning technology has made great success in fields of computer vision, natural language processing, speech recognition and so on. And machine learning model has been widely used in face recognition, automatic driving, malware detection, intelligent medical analysis and other practical tasks. In this paper, attention mechanism is proposed to combine with LSTM model to extract features in text classification. The results show that, on the one hand, LSTM + Attention can improve classification performance; on the other hand, by sorting by word weights generated by the attention layer, we find some meaningful word features, however, its recognition performance is not good. Some possible reasons were analyzed and it was found that attention mechanism sometimes misjudges wrong word features, resulting from these wrong words often appearing at the same time with meaningful word features.