AI-Driven Multimodal Mental Health Monitoring System Using Emotional, Physiological and Self-Reported Data

Abstract

Mental health challenges are on the rise, driven by modern lifestyles, workplace pressures, and social stressors. Conventional assessments often rely on self-reported questionnaires and occasional clinical visits, which can miss real-time changes and sometimes lack objectivity. To address this gap, we propose an AI-driven mental health monitoring system that uniquely integrates multiple non-invasive modalities: facial expression recognition, respiration analysis, infrared body temperature sensing, and the Questionnaire-based Model. This multimodal approach allows for continuous and holistic assessment of both emotional and physiological states. Facial expression analysis helps identify emotions such as happiness, sadness, anger, and stress; respiration monitoring captures irregular breathing patterns linked to anxiety; and temperature sensing highlights stress-induced variations. These signals are further complemented by a psychological questionnaire, which achieved the highest predictive accuracy of 93%, underscoring its effectiveness when combined with physiological cues. By uniting these different perspectives, our system not only improves detection accuracy but also reduces bias, offering a more reliable tool for early mental health intervention compared to traditional single-method approaches.