Multimodal Data Representation Based on Multi-image Concept for Immersive Environments and Online Labs Development

Abstract

In this paper we present an approach of multimodal data complex representation and processing based on the mathematical abstractions, such as aggregates and multi-images, as well as operations and relations between these mathematical objects, which are defined in the Algebraic System of Aggregates. This approach can be employed for online laboratories and immersive environments development for complex representation and processing of multimodal data which describe an object (process, phenomenon) of study. Such complex representation is resulted in obtaining a Digital Twin of the object of study that enables effective solving such tasks as modelling, prediction, classification, etc. In the paper we also propose the general architecture of an online laboratory and an immersive environment based on the presented mathematical background. A possible approach to software development for multimodal data processing is discussed in the paper as well.