When size really matters: How a single semantic feature is represented in the speech and gesture modalities

Abstract

Although some studies have demonstrated that imagistic gestures do communicate significant amounts of information none have attempted to analyze the significance of that information for the linguistic account within which it is embedded. This study focuses on the semantic dimension size and in a corpus of narratives identifies every single instance of size information and identifies whether this size information is encoded in speech, in gesture, or in speech and gesture. Crucially, it considers the judged relative importance of each instance of size information. It was discovered that high importance size information was significantly more likely to be encoded in gesture rather than in speech, whereas low importance size information was more likely to be encoded in speech rather than in gesture. This suggests that speakers may vary what information is encoded gesturally, according to its salience for the overall meaning to be conveyed. This result has major implications for the conceptualization of how gesture and speech work together in everyday talk.