Building segmentation in high-resolution remote sensing image through deep neural network and conditional random fields

Abstract

The core of building segmentation in high-resolution remote sensing image is to establish the mapping from an image feature space to a segmentation result with high dimension and strong nonlinearity. In a high-resolution remote sensing image, a building frequently emerges at any location in the entire image, thereby indicating that non-neighborhood pixels may be related to the current semantic segmentation pixel. The segmentation precision and generalization are significantly improved by adopting a Deep Neural Network (DNN) to extract the features and learn the nonlinear mapping in image segmentation. However, the non-neighborhood feature cannot be directly extracted by the DNN. This study presents an encoder-decoder deep learning architecture with ResNet and Conditional Random Field (CRF) for building semantic segmentation in a high-resolution remote sensing image to obtain high segmentation precision and reduce the obstacles from roads, staggered floors, and shadows. In the DNN, ResNet is used to establish the encoder for automatically extracting the building features, in which ResNet avoids the problems of vanishing and exploding gradient and accelerates the convergence of DNN weights. Before each convolution operation, batch normalization is adopted to normalize the sampling data and reduce the training difficulty of the DNN. Then, transposed convolution is applied to establish the decoder for reconstructing the image while segmenting the buildings. At the end of the DNN, the CRF is used to adjust the raw segmentation produced by the decoder. The value of a unary potential function in the CRF is given by the raw result of the decoder, and the pairwise potential function denotes the feature of pixel pairs in the entire image, which constructs a fully connected CRF (FCCRF). Considering that the calculation of FCCRF is considerable, a mean field algorithm is used to approximate the pairwise potential function value. Thus, convolution is used to obtain the pairwise potential function value, and a high-dimensional Gaussian filter is applied to implement the convolution operation. The mean field algorithm is implemented through an RNN mechanism. Thus, FCCRF becomes a part of the DNN, and the parameters of the CRF are trained with the encoder and decoder simultaneously. Experiments are conducted to validate the effectiveness of the proposed methodology. The remote sensing image dataset is Inria Aerial Image Labeling Dataset. A total of 4500 samples with 1000×1000×3 pixels are found in each sample, in which their resolution is 0.3 m. The typical kinds of building, such as building with order, single building with complicated roof, and building without order, are segmented through VGG, ResNet, and the proposed methodology (denoted as ResNetCRF), correspondingly. The results show that ResNetCRF overcomes the interruption of roads in which their color features are similar to the building and effectively reduces the disturbance of staggered floors and shadows. Thus, ResNetCRF obtains the optimal segmentation precision. The multi-resolution experiment demonstrates that ResNetCRF has a strong generalization under a limited range of resolution change. Accurate mapping of building segmentation is established to reduce the disturbance of roads, shadows, and staggered floors by introducing CRFs in the encoder-decoder based on ResNet to segment the building in a high-resolution remote sensing image. In the future work, we will investigate the reduction of FCCRF calculation, overcome the missing segmentation of small buildings, and reduce the segmentation errors of a building whose color feature is similar to the background without a noticeable edge.