Speech Emotion Recognition using Threshold Fusion for Enhancing Audio Sensitivity

Abstract

Speech Emotion Recognition (SER) has found applications in various fields. However, most SER studies exhibit a bias towards the text modality, which can lead to incorrect recognition when nonverbal audio features convey the primary emotional information. To address this issue, we propose a two-step solution to enhance the audio emotion sensitivity of SER models. First, we use a parallel emotional speech dataset (ESD), which contains identical speech content pronounced with different emotions, to pretrain a speech content-independent emotion recognition model, named the Audio Sensitive Network (ASN). Second, we propose a novel threshold fusion technique utilizing the Tree-structured Parzen Estimator (TPE) to optimize different thresholds for each predictive label, integrating the ASN with baseline SER classifiers. To demonstrate the efficacy of our approach, we conduct experiments on the IEMOCAP and ESD datasets. The results reveal that our novel method enhances audio sensitivity by enhancing the performance of existing SER classifiers.