RPINet: Dual-branch attention-guided fusion of remote sensing images and point clouds for urban scene segmentation

Abstract

Semantic segmentation of large-scale urban point clouds is a fundamental yet challenging task due to the complex spatial structures, massive data volume, and irregular distribution of points. Existing methods typically rely solely on 3D geometry or naively fuse 2D and 3D features, leading to limited performance in capturing both fine-grained details and global semantic consistency. In this paper, we propose RPINet, a novel Remote-Projection and Intelligent Network that effectively integrates 2D remote sensing images with 3D point cloud data for enhanced semantic segmentation. RPINet adopts a dual-branch architecture where the 3D point branch extracts spatial features using PointNet++, transformers, and graph convolutional networks, while the 2D image branch leverages a Gaussian splatting projection and CNN-based encoding to retain texture and contour information. A hybrid attention-based fusion module dynamically weights intra-modal and inter-modal dependencies, enabling deep semantic interaction between modalities. In addition, we introduce an adaptive sampling strategy and a multi-objective loss function to optimize segmentation accuracy and geometric consistency. Extensive experiments on the challenging SensatUrban dataset demonstrate that RPINet achieves state-of-the-art performance with a mean IoU of 66.5%, outperforming existing methods by a significant margin. Our model also shows strong generalization ability on unseen datasets, confirming its robustness and practical applicability to real-world urban scenarios.