Multimodal semiotic content analysis: combining quantitative large-data analysis with qualitative semiotic coding

Abstract

This article proposes a mixed-method framework for multimodal semiotic content analysis that systematically integrates qualitative semiotic coding with quantitative large-scale data analysis. Building on earlier frameworks by Bell and Milic (‘Goffman’s gender advertisements revisited: combining content analysis with semiotic analysis’, in Visual Communication, 2002) and Serafini and Reid (‘Multimodal content analysis: expanding analytical approaches to content analysis’, in Visual Communication, 2023), the authors propose a more empirically rigorous method to conduct quantitative analysis. They highlight the limitation of frequency counts through percentage comparisons and advocate for using proper statistical methods for identifying meaningful co-patterning of semiotic resources. Their approach also emphasizes the importance of transparent coding practices, detailed codebooks, pilot coding and intercoder reliability testing to enhance replicability and validity. The framework is illustrated through a case study of 277 depression-related internet memes. Results reveal significant associations across depression themes and semiotic resources of process types and social interaction, highlighting recurring communicative strategies such as infographical, emotional, dialogical and suicidal gaze memes. By demonstrating how multimodal choices co-pattern to construct meaning, this framework offers a practical, statistically grounded method for analysing complex visual and textual datasets beyond descriptive counts.