Hierarchical Dual Attention-Based Recurrent Neural Networks for Individual and Group Activity Recognition in Games

Abstract

We study the problem of simultaneously recognizing complex individual and group activities from spatiotemporal data in games. Recognizing complex player activities is particularly important to understand game dynamics and user behavior having a wide range of applications in game development. To do so, we propose a novel framework by developing a hierarchical dual attention RNN-based method that leverages feature and temporal attention mechanisms in a hierarchical setting for effective discovery of activities using interactions among individuals. We argue that certain activities have dependency on certain features as well as on temporal aspects of the data which can be leveraged by our dual-attention model for recognition. To the best of our knowledge, this work is the first to address activity recognition using spatiotemporal data in games. In addition, we propose using game data as a rich source of obtaining complex group interactions. In this paper, we present two contributions: (1) two annotated game datasets that consist of individual and group activities, (2) our proposed framework improves the state-of-the-art recognition algorithms for spatiotemporal data by experiments on these datasets.