Continuity, Co-reference, and Inference in Visual Sequencing

Abstract

To understand narratives, comprehenders need to establish to what extent various expressions refer to the same entity, known as co-reference. Research in linguistics has long recognized this fundamental aspect of meaning-making, and perhaps as such, an increasing amount of works have attempted to extend this knowledge to other narrative expressions, such as visual storytelling. However, these applications of semantic and discourse models to visual narratives do not sufficiently account for how different visual signals may be reconciled into a common entity. This core question remains unanswered even in theories characterizing meaning relations for visual sequences in comics specifically. Therefore, this paper proposes a formal theory of visual co-reference that accounts for these issues. We present a model of constraints that describes visual co-reference based on varying correspondences between form (graphic input) and meaning (conceptual structure), including subsequent inferences about temporal and spatial relations across panels and more complex structures of visual discontinuity. This theory is in line with both corpus and psychological findings of visual narratives, and thus presents a robust framework for analyses of visual co-reference.