Gaze-aware graph convolutional network for social relation recognition

Abstract

Social relation, as the basic relation in our daily life, is vital for social action analysis. However, how to learn the social feature between people is still not tackled. In this work, we propose a gaze-aware graph convolutional network (GA-GCN) for social relation recognition, which targets discovering the context-aware social relation inference with gaze-aware attention. To predict the gaze direction, we apply a convolutional network trained with gaze direction loss. Then, we build a graph convolutional inference module, which is a two-stream graph inference with both gaze-aware attention and distance-aware attention. The attention can pick up relevant context objects for context-aware representation. We further introduce additional scene features and construct a multiple feature fusion module, which can adaptively learn social relation representation from both scene feature and context-aware feature. Extensive experiments on the PISC and the PIPA datasets demonstrate that our GA-GCN can find interesting contextual objects and achieves state-of-the-art performances.