Opinion mining on social media transit tweets using text pre-processing and machine learning techniques

Abstract

Capturing public insights related to transit systems in social media has gained huge popularity presently. The regional transportation agencies use social media as a tool to provide information to the public and seek their inputs and ideas for meaningful decision making in transportation activities. This exploratory study attempts to gauge the impact of social media use in transportation planning that in turn would help transportation administration in identifying the day-to-day challenges faced by the customers and to suggest a suitable solution. This paper presents the effect of pre-processing techniques on transit opinion analysis to improve the performance. Performance of different pre-processing methods namely stop word removal, stemming, lemmatization, negation handling and URL removal using feature representation models namely TF-IDF with unigram, TF-IDF with bigram on three feature selection techniques including information gain, standard deviation and chi-square on social media transit rider’s opinion is carried out. The experimental results are evaluated using four different classifiers such as Support vector machine, Naïve Bayes, Decision Tree, K-Nearest Neighborhood in terms of accuracy, precision, recall, and f-measure. On analyzing the social media related transit opinion data, it is observed that pre-processing with bigram technique performs better than the other approaches specifically with Support Vector Machine and Naïve Bayes.