Transformer-GAN hybrid architecture for cross-modal virtual-real alignment in intelligent manufacturing system design

Abstract

Smart manufacturing systems are undergoing deep digitalization and networking. However, two core bottlenecks persist: (1) Heterogeneous multimodal data integration (vision, RF, vibration), and (2) Virtual-real system collaboration barriers. These challenges directly restrict the evolution towards high-level intelligence. Current multimodal learning technologies face key challenges such as insufficient modeling of physical constraints in industrial data, low reliability of generative adversarial network (GAN) cross-modal mapping, and limited efficiency of Transformer architecture deployment at the edge. Existing virtual-real alignment methods, lacking the ability to model dynamic spatiotemporal correlations, struggle to adapt to flexible manufacturing requirements. This study proposes a cross-modal virtual-real alignment training framework based on a Transformer-GAN hybrid architecture. It adaptively integrates features from multiple sources such as vision, radio frequency, and vibration through a dynamic weight allocation mechanism. It combines a progressive training strategy to balance the collaborative optimization of virtual simulation and physical data, and designs a lightweight model compression scheme to achieve efficient deployment on edge devices. Experiments show that this framework significantly improves the quality of multimodal generation and the accuracy of virtual-real trajectory alignment. It breaks through the limitations of traditional methods in terms of industrial-level accuracy, real-time performance, and economic indicators, significantly improving multimodal generation quality (FID = 18.9 vs. CycleGAN’s 42.7) and virtual-real alignment accuracy (VAM-IoU = 88.3%). Edge deployment achieves 32 ms latency on Jetson AGX with 65.2% model compression, enabling real-time control in welding and assembly scenarios. The framework’s industrial efficacy is validated through a 23.8% OEE improvement and 40% annual cost reduction in automotive production lines.