A Multi-modal Seq2seq Chatbot Framework

Abstract

The pandemic has forced young people to stay away from school and friends, complete online learning at home and live at home. Therefore, various mental illnesses such as anxiety and depression occur more frequently. Chatbot is a communication method that is more acceptable to young people. This paper proposes a multi-modal chatbot seq2seq framework, which divides the mental state of young people into different types through multi-modal information such as text and images entered by users in the chatbot. This model combines image description and text summarization modules with the attention mechanism in a multi-modal model to control related content in different modalities. Experiments on multi-modal data sets show that this method has 70% average accuracy and real users who use this system also believe that this method has good judgment ability.