The Rise of Multimodal Tutors in Education: Insights from Recent Research

Abstract

This chapter describes the insights derived by the design and development of the Multimodal Tutor, a system that uses artificial intelligence for providing digital feedback and to support psychomotor skills acquisition. In this chapter, we discuss the insights which we gained from eight studies: (1) an exploratory study combining physiological data and learning performance (Learning Pulse); (2) a literature survey on multimodal data for learning and a conceptual model (the Multimodal Learning Analytics Model); (3) an analysis of the technical challenges of Multimodal Learning Analytics (the Big Five Challenges); (4) a technological framework for using multimodal data for learning (the Multimodal Pipeline); (5) a data collection and storing system for multimodal data (the Learning Hub); (6) a data annotation tool for multimodal data (the Visual Inspection Tool); (7) a case study in Cardiopulmonary Resuscitation training (CPR Tutor) consisting of a feasibility study for detecting CPR mistakes; and (8) a real-time feedback study.