Multimodal Strategy to Defend Mobile Devices Against Vishing Attacks

Abstract

In the landscape of Vishing calls, it's crucial to determine if users are unknowingly falling prey to fraudsters' tactics, potentially leading to fraud. This paper introduces the Multimodal Vishing Threat Detection (MmVTD) framework, designed to counter Vishing threats and issue alerts on potential fraudulent activities leveraging three key data modalities: call transcripts, sequences of mobile screenshots, and extracted Optical Character Recognition(OCR) text. The novelty of our approach lies in utilization of transformer encoders to build the learnable context for each modality, with a decoder generating warning content while an MLP head classifies the current call as Vishing or Safe at any given instance. This innovative architecture enhances the model's ability to analyze the diverse data sources and generate effective warning in real-time. Furthermore, the MmVTD framework offers robust protection through multimodal analysis, providing real-time alerts when user action suggest susceptibility fraud. Finally, the MmVTD model achieved an impressive 94.44% accuracy in identifying Vishing and Safe calls on a dedicated dataset. Additionally, it attained a BLEU score of 0.583 in generating cautionary messages, effectively deterring users from potential harmful actions.