UrduThreat@ FIRE2021: Shared Track on Abusive Threat Identification in Urdu

Abstract

With the growth of spread and importance of social media platforms, the effect of their misuse became more and more impactful. This shared task address the task of abusive and threatening language detection in Urdu language that has more than 230 million speakers worldwide. We presented two datasets: (i) Abusive and Non-Abusive language, (ii) Threatening and Non-Threatening language. The abusive dataset contains 1,187 tweets categorized as Abusive and 1,213 as Non-Abusive and the threatening dataset contains 4,929 tweets categorized as Non-Threatening and 1,071 as Threatening. In this shared task, 21 teams registered for participation from six countries (India, Pakistan, China, Malaysia, United Arab Emirates, Taiwan), 10 teams submitted their runs for Subtask A - Abusive Language Detection, 9 teams submitted their runs for Subtask B - Threatening Language detection, and seven teams submitted their technical reports. We provided one baseline system for Subtask A and three baseline systems for Subtask B. The best performing system achieved an F-score value of 0.88 for Subtask A and 0.545 for Subtask B. For both subtasks, m-Bert based transformer models showed the best performance.