Code switch point detection in arabic

Abstract

This paper introduces a dual-mode stochastic system to automatically identify linguistic code switch points in Arabic. The first of these modes determines the most likely word tag (i.e. dialect or modern standard Arabic) by choosing the sequence of Arabic word tags with maximum marginal probability via lattice search and 5-gram probability estimation. When words are out of vocabulary, the system switches to the second mode which uses a dialectal Arabic (DA) and modern standard Arabic (MSA) morphological analyzer. If the OOV word is analyzable using the DA morphological analyzer only, it is tagged as DA, if it is analyzable using the MSA morphological analyzer only, it is tagged as MSA, otherwise if analyzable using both of them, then it is tagged as both. The system yields an F β = 1 score of 76.9% on the development dataset and 76.5% on the held-out test dataset, both judged against human-annotated Egyptian forum data. © 2013 Springer-Verlag Berlin Heidelberg.