gUI: Specifying Complete User Interaction

Abstract

With emotion described as “the organism’s interface to the world outside” (Scherer [45]), there has been great interest in the role of emotion in speech and gestures in making Human—Virtual Human interfaces more effective. Miller [33] suggests that only 7% of a message is sent through words: the remainder is sent through facial expressions (55%) and vocal intonation (38%). Therefore in both analysis of human conversations and in the synthesis of Virtual Humans, expressive emotion and gestures need to be catered for to ensure that the intent of the message is not lost. A radical paradigm change occurred in going from text entry to the mouse-pointer concepts of a Graphical User Interface. In a similar way, it is now necessary for a total user input paradigm, adding video and audio input to the existing methods, to become the predominant Computer Human Interaction (CHI) of the future. This complete interaction is referred to as a gestalt user interface: an interface that should be reactive to, and proactive of, the perceived desires of the user through emotion and gesture. The formal specification, development, implementation, and evaluation of a gestalt User Interface (gUI) language is necessary to provide a stable, consistent base for future research into multi-modal Human interfaces in general, and specifically to Embodied Character Agents. This chapter details some early research on language design and also an implementation evaluation.