Fusion of XLNet and BiLSTM-TextCNN for Weibo Sentiment Analysis in Spark Big Data Environment

Abstract

This article proposes a Weibo sentiment analysis method to improve traditional algorithms' analysis efficiency and accuracy. The proposed algorithm uses deep learning in the Spark big data environment. First, the input data are converted into dynamic word vector representations using the Chinese version of the XLNet model. Then, dual-channel feature extraction is performed on the data using TextCNN and BiLSTM. The proposed algorithm uses an attention mechanism to allocate computing resources efficiently and realizes feature fusion and data classification. Comparative experiments are conducted on two public datasets under identical experimental conditions. In the NLPCC2014 and NLPCC2015 datasets, the proposed model improves the precision and F1 metrics by at least 4.26% and 2.64%, respectively. In the weibo_senti_100k dataset, the proposed model improves the precision and F1 metrics by at least 4.66% and 2.69%, respectively. The results indicate that the proposed method has better sentiment analysis and prediction abilities than existing methods.