Relatable clothing: Soft-attention mechanism for detecting worn/unworn objects

Abstract

We have identified a need in visual relationship detection and biometrics related research for a dataset and model which focuses on person-clothing pairs. Previous to our Relatable Clothing dataset, there were no publicly available datasets usable for ''worn'' and ''unworn'' clothing detection. In this paper we propose a novel visual relationship model architecture for ''worn'' and ''unworn'' clothing detection that makes use of a soft attention mechanism for feature fusion between a conventional ResNet backbone and our novel person-clothing mask feature extraction architecture. The best proposed model achieves 98.62% accuracy, 99.50% precision, 98.31% recall, and 99.14% specificity on the Relatable Clothing dataset, outperforming our previous iterations. We release our models which can be found on the Relatable Clothing GitHub repository (https://github.com/th-truong/relatable_clothing) for future research and applications into detecting and analyzing person-clothing pairs.