Modelling multimodal interaction: A theory-based technique for design analysis and support

Abstract

This tutorial introduces participants to a way of thinking about how users mentally represent different facets of their interactions with advanced interfaces involving multimodal information. The approach is theory-based. It assumes that human information processing deals with nine different types of mental representation. General rules of combination and decomposition of constituents, basic units and the superordinate organisation of all types of representation will be presented. The first part of the tutorial introduces the basic theoretical ideas. The second part links this basic theory into the domain of HCI and illustrates how core concepts can be used to analyse the course of interactions with static visual displays, dynamic visual displays and multimodal displays. In the final part a handbook will be introduced describing a simple technique and an associated notation for representing the static and dynamic properties of visual information, task structures and acoustic information.