Smart beaker based on multimodal fusion and intentional understanding

Abstract

In the current simulation experiment system, the experimental design of single mode is less interactive and less accurate. In order to solve this problem, this paper proposes an experimental interaction kit based on sound and sensor, and designs a multimodal fusion and intent understanding algorithm. Firstly, the method of multi-sensor signal extraction and speech feature extraction is introduced. Then, based on the results obtained by the two methods, an algorithm based on decision-level fusion is studied, which solves the problem of perception of user's operation intention in virtual chemistry experiments. Finally, the usability of the multimodal intent understanding algorithm proposed in this paper is verified by designing a complete chemical experiment system. Experiments show that the multimodal intent understanding algorithm based on sensor and speech input is due to a single modality in terms of interactivity and accuracy, and the physical interaction suite designed in this paper greatly improves the intelligence and interactivity of the system.