Selecting NLP Classification Techniques to Better Understand Causes of Mass Killings

Abstract

We perform an analysis of SVM, BERT, and Longformer NLP tools as applied to large volumes of unclassified news articles given small volumes of labeled news articles for training. Analysis of the target machine learning tools is performed through a case study of global trigger events; specifically triggers of state-led mass killings. The goal of the case study is to draw relationships from the millions of machine classified articles to identify trends for the prediction and prevention of future mass killing events. In this paper we focus on the classification one specific trigger, coups, in order to glean insight into the accuracy and complexity of our SVM, BERT, and Longformer models. This study centers on classifying which news articles contain uniquely defined coup events and the temporal placement of those articles. Our performance analysis centers on the comparison of multiple accuracy metrics as applied to specific subsets of the corpus. We also demonstrate that raw accuracy scores are insufficient to fully understand the quality of classification required for specific target use cases.