An italian multimodal corpus: The building process

Abstract

During the design of multimodal interaction environments, the use of a corpus of multimodal sentences is very important in order to achieve various tasks of multimodal interaction. In last decade, several researchers addressed the creation of multimodal corpora for English, French, and various other languages. However, from the analysis of these multimodal corpora, there clearly is a lack of multimodal corpora for Italian. This paper describes the building process of an Italian multimodal corpus. Starting from the manual analysis of multimedia dialogues, this process extracts different multimodal data, i.e. speech and gestures, which are used to generate grammar rules and to train the multimodal interpreter in order to set the framework for the multimodal corpus building. Following that, the set framework is used to annotate semi-automatically multimodal information, such as syntactic roles and semantics, on new dialogues to be included in the corpus.