A Personalised Approach to Audiovisual Humour Recognition and its Individual-level Fairness

Abstract

Humour is one of the most subtle and contextualised behavioural patterns to study in social psychology and has a major impact on human emotions, social cognition, behaviour, and relations. Consequently, an automatic understanding of humour is crucial and challenging for a naturalistic human-robot interaction. Recent artificial intelligence (AI)-based methods have shown progress in multimodal humour recognition. However, such methods lack a mechanism in adapting to each individual's characteristics, resulting in a decreased performance, e.g., due to different facial expressions. Further, these models are faced with generalisation problems when being applied for recognition of different styles of humour. We aim to address these challenges by introducing a novel multimodal humour recognition approach in which the models are personalised for each individual in the Passau Spontaneous Football Coach Humour (Passau-SFCH) dataset. We begin by training a model on all individuals in the dataset. Subsequently, we fine-tune all layers of this model with the data from each individual. Finally, we use these models for the prediction task. Using the proposed personalised models, it is possible to significantly (two-tailed t-test, p < 0.05) outperform the non-personalised models. In particular, the mean Area Under the Curve (AUC) is increased from.7573 to.7731 for the audio modality, and from.9203 to.9256 for the video modality. In addition, we apply a weighted late fusion approach which increases the overall performance to an AUC of.9308, demonstrating the complementarity of the features. Finally, we evaluate the individual-level fairness of our approach and show which group of subjects benefits most of using personalisation.