A multimodal reference resolution approach in virtual environment

Abstract

This paper presents a multimodal reference resolution approach in virtual environment, which is called RRVE. Based on the relationship between cognitive status and reference, RRVE divides the objects into four status hierarchies including pointing, in focus, activated, extinct, and step by step it processes multi-modal reference resolution according to current status hierarchy. Also it defines a match function to compute the match probability of referring expression and potential referent, and describes the semantic signification and temporal constraints. Finally, sense shape is used to deal with the pointing ambiguity, which helps the user to interact precisely in immersive virtual environment. © Springer-Verlag Berlin Heidelberg 2006.