Scene graph generation via convolutional message passing and class-aware memory embeddings

Abstract

Detecting visual relationships between objects in an image still remains challenging, because the relationships are difficult to be modeled and the class imbalance problem tends to jeopardize the predictions. To alleviate these problems, we propose an end-to-end approach for scene graph generations. The proposed method employs the ResNet as the backbone network to extract the appearance features of the objects and relationships. An attention based graph convolutional network is exploited and modified to extract the contextual information. Language and geometric priors are also utilized and fused with the visual features to better describe the relationships. At last, a novel memory module is designed to alleviate the class imbalance problem. Experimental results demonstrate the validity of our model and our superiority compared to our baseline technique.