A Systems Theory of Multimodality

Abstract

This paper offers a tentative theory of how to approach the analysis of multimodal “texts” (using the word loosely for any integrated set of symbols that resemble written texts in communicative power).  Much work on multimodality is new and many key issues are not yet broached, let alone settled.  The theory I produce treats modes as semiotic systems (sets of social conventions about meaning) and multimodality as a system of systems.  A theory in a new domain helps us understand what we take the “things” we are going analyze to be (our “ontology”).  In turn, our choice of “things” determines the sorts of tools we will need for analysis and the sorts of questions we will ask, a number of which are delineated in this paper.  The theory developed here is based on the correspondences between linguistic systems and non-linguistic modal systems.  The paper closes with a discussion of the implications of the approach I propose for work on design and learning.