Temporal, structural, and pragmatic synchrony between intonation and gesture

Abstract

This paper explores the interaction between intonation and gesture, noting temporal, structural, and pragmatic synchrony. Videotapes of subjects conversing freely were annotated for intonation according to ToBI (Beckman and Elam 1997), and for gesture according to Kendon (1980) and McNeill (1992). The time-stamped annotations were analyzed statistically, as well as visually in the Anvil tool (Kipp 2001), which allows time-aligned viewing of videos with their annotations. Alignments were investigated between three levels of intonational units and four levels of gestural units. The intonational units were, from smallest to largest, pitch accents, intermediate phrases, and intonational phrases. The gestural units, again from smallest to largest, were apices of strokes, gesture phases, gesture phrases, and gesture units. Of these possible combinations, two pairs aligned. Apices clearly aligned with pitch accents, and gesture phrases tended to align with intermediate phrases. The existence of intermediate phrases in English has been the subject of some debate (Ladd 2008), and this paper suggests that a probable gestural correlate to intermediate phrases provides independent evidence for their existence. In addition, intonation and gesture were found to perform simultaneous complementary pragmatic functions. This temporal, structural, and pragmatic synchrony between the two channels reinforces the claim that speech and gesture are two surface forms of the same underlying and emerging cognitive content.