Identifying User Interests and Habits Using Object Detection and Semantic Segmentation Models

Abstract

The article describes a software pipeline for identifying and classifying the interests of users in social networks using modern models and deep learning methods. The developed program is able to detect the presence of bad habits (smoking, alcohol), a sporting lifestyle, as well as determine the user's addiction to travel by an available set of photos. The software includes modules that implement deep learning algorithms for the object detection and semantic segmentation of images using the Cascade-R-CNN and DeepLabv3+ models, and the module for converting annotations of the images from COCO, ImageNet, OpenImagesV6 datasets and manually labeled images to the unified format. The models were trained on the created original datasets which include 90200 photos in total. The accuracy of the developed models is from 83.7% up to 86.6% mAP for object detection depending on a specific category of objects and 78.4% pixel accuracy for segmentation.