Research on Virtual Try-on Immersive Recommendation Based on Multimodal Large Model

Abstract

Traditional e-commerce recommendation systems often struggle with processing multimodal data, addressing cold start challenges, and delivering immersive user experiences. To overcome these limitations, this study introduces a novel framework—M2VTT-Rec (Multimodal Model-driven Virtual Try-on Recommendation)—which aims to transform personalized recommendations in online retail. The M2VTT-Rec framework deeply incorporates users’ multimodal interactions during virtual try-on sessions—including real-time imagery, biometric signals, facial expressions, and voice cues—alongside multimodal product knowledge, such as 3D product models and design semantics.The system features a cutting-edge module for sensing and encoding try-on data across modalities, coupled with a fine-grained preference alignment network and a bidirectional reinforcement learning mechanism that captures user visual inclinations and evolving interests. Furthermore, a context-sensitive recommendation generator is designed to produce rich multimodal outfit suggestions and interpretable narrative explanations.Empirical evaluation using a purpose-built multimodal virtual try-on dataset shows that M2VTT-Rec achieves superior performance over the baseline ALS model on key metrics like Recall@10 and NDCG@10, with especially strong results in various cold start scenarios. Additionally, it markedly improves the average session duration and boosts user engagement and satisfaction across user segments with different purchasing habits. Ablation studies further validate the essential roles of multimodal implicit feedback representation and reinforcement learning in achieving these outcomes. Overall, this research presents a forward-looking approach for applying multimodal large models in vertical domains such as e-commerce, paving the way for more intelligent, interactive recommendation systems.