Facial Emotion Recognition using Video and Audio

Abstract

This research project adopts a holistic strategy for comprehending multimedia content, synergizing advanced techniques in video and audio analysis. Video analysis employs state-of-the-art Convolutional Neural Networks (CNNs), specifically ResNet50V2 and ResNet152V2, renowned for their excellence in image and video classification. Simultaneously, audio analysis incorporates a Multi-Layer Perceptron (MLP) classifier, a custom-built CNN tailored for audio processing, and Support Vector Machine (SVM) models. ResNet architectures are harnessed to extract high-level features from video frames, providing a robust foundation for visual content understanding. The audio analysis, employing an MLP classifier and a self-designed CNN, captures intricate audio patterns, enhancing overall capabilities. The SVM model further contributes to audio content understanding. The integration of extracted features from both modalities enables a holistic interpretation, enhancing the system's ability to recognize complex patterns in multimedia data. Experimental results validate the approach, demonstrating improved accuracy and robustness. This collaboration between ResNet architectures and audio classifiers presents promising applications in content recommendation and emotion-aware multimedia processing. The study contributes to the evolving field of multimodal deep learning, paving the way for further advancements in interpreting diverse multimedia content.