Advancing Audio Processing and Emotion Recognition through Deep Learning Techniques

Abstract

Speaker diarization is essential in audio processing, distinguishing and attributing speech segments to individual speakers in applications like speech recognition, customer service analytics, social media monitoring, and broadcast transcription. Despite advancements, challenges such as overlapping speech, varying acoustic environments, and noise resilience persist. This work proposes a deep learning-based approach using a pre-trained transformer model to handle multiple tasks, including feature extraction, Voice Activity Detection (VAD), segmentation, and speaker change detection. The system employs the ECAPA-TDNN model for speaker embeddings, followed by Agglomerative Hierarchical Clustering (AHC) for speaker labeling. Additionally, it performs sentiment analysis by converting segmented speech into text and applying a lexicon-based model. The implementation supports real-time audio recording, speaker segmentation, sentiment analysis, and visualization. Tested on multi-speaker recordings, it achieved 93.4% transcription accuracy while reducing computational costs, making it more feasible for standard hardware compared to traditional resource-intensive methods. By integrating speaker diarization and sentiment analysis within a unified framework, this research contributes to real-time, speaker-aware applications in various domains.