Multimodal use of an upper-level event ontology

Abstract

We describe the ongoing development of a lexically-informed, upper-level event ontology and explore use cases of the ontology. This ontology draws its lexical sense distinctions from VerbNet, FrameNet and the Rich Entities, Relations and Events Project. As a result, the ontology facilitates interoperability and the combination of annotations done for each independent resource. While this ontology is intended to be practical for a variety of applications, here we take the initial steps in determining whether or not the event ontology could be utilized in multimodal applications, specifically to recognize and reason about events in both text and video. We find that the ontology facilitates the generalization of potentially noisy or sparse individual realizations of events into larger categories of events and enables reasoning about event relations and participants, both of which are useful in event recognition and interpretation regardless of modality.