Vision and Radar Multimodal Aided Beam Prediction: Facilitating Metaverse Development

Abstract

The metaverse requires enhanced communication rates and increased capacity, which can be attained by utilizing millimeter wave (mmWave) and terahertz (THz) communication systems along with large-scale antenna arrays. However, these systems come with a considerable beam training overhead. To address this challenge, this study proposes a novel multimodal deep learning framework based on 3D convolutional transformers for sensor-assisted beam prediction. Our approach utilizes both vision and radar data, resulting in quick and precise beam prediction. Our proposed scheme demonstrates more than 78% top-3 beam prediction accuracy in four different communication scenarios. Furthermore, the total prediction accuracy of our proposed framework is 85.6%, which is nearly 10% higher than using only single-sensory data. Our proposed solution effectively reduces beam training overhead and provides reliable communication support for high-mobility environments.