Multimodality in Language and Speech Systems — From Theory to Design Support Tool

Abstract

Advances in information technologies and telecommunications are making available a very large number of possible input/output modality combinations which are potentially useful for the expression and exchange of information in human-human-system interaction (HHSI). However, standard or shared foundations consisting of all available and possible elementary, or unimodal, input and output modalities is still lacking. A common representation format for describing the relevant properties of individual unimodal modalities is lacking as well. Ideally, such foundations could enable designers of multimodal interaction to select an appropriate set of modalitie s for an application to be designed, once its information representation and exchange requirements have been specified. This chapter proposes Modality Theory-based solutions and discusses alternative approaches to major parts of this practical problem of modality selection. The solutions cover the generation, at descending levels of abstraction, of taxonomies of unimodal input and output modalities from basic properties in the media of graphics, acoustics and haptics. The result is a practical ‘designer's t oolbox’ of unimodal modalities. A first common representation format for unimodal modalities is presented. The taxonomy and the unimodal modalities as represented in the common format are then brought to the test by addressing the complex issue of when (not) to use speech in practical applications. Finally, empirical and theoretical approaches to the combinatorial explosion of modality combinations in multimodal systems are discussed.