Multimodal fNIRS-EEG classification using deep learning algorithms for brain-computer interfaces purposes

Abstract

The development of brain-computer interface (BCI) systems has received considerable attention from neuroscientists in recent years. BCIs can serve as a means of communication and for the restoration of motor function for patients with motor disorders. An essential part of the design of a BCI is correctly classifying the brain signals, historically collected using electroencephalography (EEG). However, recent studies have shown more robust classification results when EEG is combined with other neuroimaging methods such as fNIRS. Conventional classification methods need a priori feature preprocessing to train the model; such feature selection is a difficult and heavily studied problem. By using deep neural networks (DNN), in which recordings can be fed directly to the algorithm for training, we avoid the need for feature selection. In this study, the capabilities of DNNs in the classification of the hybrid EEG-fNIRS recordings of motor imagery (MI) and mental workload (MWL) tasks are investigated. A five-layer fully connected network is used for classification. This study makes use of two open-source meta-datasets collected at the Technische Universitat Berlin. The first dataset includes brain activity recordings of 26 healthy participants during three cognitive tasks: (1) n-back (0-, 2- and 3-back), (2) discrimination/selection response task (DSR) and (3) word generation (WG) tasks. The second dataset, motor imagery, consists of left and right-hand motor imagery tasks, each for 29 healthy participants. Our results show that classification accuracy is considerably higher for multimodal recordings when compared to EEG or fNIRS recordings alone. The proposed algorithm improves classification performance relative to a conventional support vector machine (SVM), reaching 90% average accuracy for both tasks, 8% higher than SVM performance. These results demonstrate the feasibility of achieving strong classification performance using multimodal BCI and deep learning.