Getting Around the Semantics Challenge in Hateful Memes

Abstract

Social media has evolved into a forum for people to express their thoughts and ideas on a wide range of topics. With this, there has been a correlated rise in hate and inflammatory speeches against individuals and organisations—sometimes with severe consequences. Methods to classify ‘hate’ that is propagated through multi-modal media, such as memes (contains image and text), fail to capture the meaning by comprehending both the modes(i.e. image and text). While the text itself may not be hateful, images are utilised to lend additional context to the words to subtly imply and convey hatred. On the Facebook Hate Meme Dataset, specifically curated for conveying hate implicitly in memes (further complicated by the use of ‘benign confounders’), the baseline established by Facebook with a state-of-the-art visual-linguistic model such as VilBERT is 64.73%. On the same dataset, our work beats the state-of-the-art baseline models by nearly 5% using an effective fusion of the semantics of both the text and image.