Identification of age and gender in pinterest by combining textual and deep visual features

Abstract

In social media users share a lot of content, such as comments, news, photos, videos, etc. This information can be used by automated systems to segment the users to provide them with specific recommendations or focused content. One of the most popular way to segment the users is by age and gender. Nevertheless, such demographic variables are frequently hidden, and thus becomes useful to indirectly infer them. Commonly, these variables are learned using the text comments the users publish, analyzing the style of writing or frequency of words. In this paper, we present a study of several machine learning models that employ user generated images and text trying to exploit both types of information to infer the age and gender for Pinterest users. We experiment with the models using a dataset composed of 548,761 pins, posted by 264 users. Each pin is a combination of an image and a short comment. We transformed the images to a deep visual representation using the pretrained convolutional neural network ResNet-50, and transformed the comments using the tf-idf method. We compare the models among them and between the types of information using different performance metrics. Our experiments show interesting results and the viability of employing the user generated image and text content to characterize users.