Assessing Mental Health During Covid-19 Lockdown: A Smartphone-Based Multimodal Emotion Recognition Approach

Abstract

The large-scale lockdowns that occurred during the coronavirus pandemic have caused a severe public mental health crisis. Consequently, a fast and effective method for assessing the mental state of the public is essential. As artificial intelligence technology advances, smartphone-based multimodal emotion recognition technology can provide potential solutions to this problem. Fifty-five participants who lived in lockdown areas participated in this study. We extracted facial expressions, acoustic features, and text information from five-minute self-recorded videos - as well as participants' cardiovascular information - using smartphone photoplethysmography (PPG) technology, which converted videos of their fingertips into heart-rate information. Support vector machine (SVM) and random forest (RF) algorithms were used for each modality - as well as in the final fusion stage - to determine the best multi-modality prediction model for mental health. Our results showed that the prediction accuracy of various mental health indicators using the single-modality models was between 0.31 and 0.52. However, the multi-modality model provided better, more stable results. This study validated the feasibility of using smartphones in mental health assessments of residents under lockdown. Our data also supported the use of multi-modality models and confirmed their superiority in emotion recognition.