Sensor and Feature Level Fusion of Thermal Image and ECG Signals in Recognizing Human Emotions

Abstract

Recent studies on recognition of various emotion labels concentrated on speech signals, text, visual images and anatomical variables. The proposed system combines the features of ECG which are extracted using empirical mode decomposition and features of thermal images which are extracted from Gray Level Co-occurrence Matrix (GLCM) viz energy, contrast, homogeneity and correlation. ECG is acquired from AD8232 module and thermal images from FLUKE TiS20. Data of ECG and thermal images are acquired simultaneously from a subject and database consists of data from 40 subjects in age group of 20 years to 40 years from Hassan, Karnataka, India. Here different labels of emotions have been classified based on K-nearest neighbor decision rule. This system yielded highest accuracy for disgust and lowest for anger using ECG and highest accuracy for disgust and surprise and least for sad.