On Soft-Biometric Information Stored in Biometric Face Embeddings

Abstract

The success of modern face recognition systems is based on the advances of deeply-learned features. These embeddings aim to encode the identity of an individual such that these can be used for recognition. However, recent works have shown that more information beyond the user's identity is stored in these embeddings, such as demographics, image characteristics, and social traits. This raises privacy and bias concerns in face recognition. We investigate the predictability of 73 different soft-biometric attributes on three popular face embeddings with different learning principles. The experiments were conducted on two publicly available databases. For the evaluation, we trained a massive attribute classifier such that can accurately state the confidence of its predictions. This enables us to derive more sophisticated statements about the attribute predictability. The results demonstrate that the majority of the investigated attributes are encoded in face embeddings. For instance, a strong encoding was found for demographics, haircolors, hairstyles, beards, and accessories. Although face recognition embeddings are trained to be robust against non-permanent factors, we found that specifically these attributes are easily-predictable from face embeddings. We hope our findings will guide future works to develop more privacy-preserving and bias-mitigating face recognition technologies.