Continuous Temporal Contiguity Narration-Animation for Learning in Virtual Reality

Abstract

Background: Temporal contiguity between visual and verbal (narrated) information could strongly influence learning from multimedia animations, especially in the case of a rich visual context such as Virtual Reality. Compared to spatial contiguity, less research has been carried out on the impact of the time lag, for example, the de-synchronisation, between visual (pictures) and verbal (text) information during animation-based learning and mainly on adults. Further, the results of this research were mixed, especially in the case of continuous temporal contiguity. The material conventionally used was composed of short, schematic animations and not of a rich, visual, learning context and no studies have been carried out on temporal contiguity in virtual reality. Objectives: In order to help satisfy this need for research, we used a multimedia lesson, in VR (desktop) about organic matter decomposition in a forest environment to investigate the influence of the (de)-synchronisation of, for example, time lag between, narrated text and dynamic pictures in rich PC-VR video. The theoretical framework of the study involved a consistent combination of the Cognitive Theory of Multimedia Learning (CTML, Mayer 2014) and of the Time-Based Resource-Sharing model of working memory (TBRS, Barrouillet and Camos 2021). Method: In an experimental pre-post-test learning paradigm, the effects of five new temporal contiguity conditions on various recall measures, comprehension, and also new episodic memory tasks were tested on 227 junior secondary school participants (second year: 12 years old). They were invited to study an on-screen Virtual Reality lesson on the European forest soil decomposition mechanism. The delays between the narrated words and the corresponding animations were: −6, −2, 0, +2, +6 s, respectively. Prior knowledge, working memory capacity, and spatial abilities were controlled. Results and Conclusions: Temporal gaps, even −2 s had a significant negative effect on recall and comprehension. Performances were influenced by working memory capacity. However, the episodic memory task results suggested that text-picture integration with long text segments was greater than previous literature has shown. Learners were relatively aware of time lags. Implications: In the future, precise time-locked correspondence between spoken text and dynamic pictures might turn out to be of high importance for immersive VR, because of the high degree of interactivity, for example, user control of the environment dynamics. These results are consistent with the CTML and the TBRS models, and they suggest that it would be worthwhile to incorporate episodic memory measures into the CTML.