Multimodal Deep Learning

Abstract

Multimodal deep learning has gained significant attention and shown great promise in various domains, including medical, manufacturing, Internet of Things (IoT), remote sensing, and urban big data. This chapter provides an overview of neural network-based fusion techniques in multimodal deep learning. The advantages of deep learning over conventional shallow learning methods are discussed, highlighting its ability to learn both inter- and intra-modality representations with minimal preprocessing and implicit dimensionality reduction. The chapter explores different fusion methods, including early fusion, late fusion, and intermediate fusion, and discusses their capabilities and limitations. It also examines various objectives used in late fusion, such as reconstruction error, correlation-based objectives, and semantic alignment. The challenge of avoiding negative transfer in multimodal learning is addressed, and regularization objectives and training approaches are explored. Overall, this chapter serves as a comprehensive guide to multimodal deep learning and its fusion techniques, offering insights into their applications and potential for future research.