Image-Based 3D Shape Estimation of Wind Turbine from Multiple Views

Abstract

This paper addresses the problem of reconstructing depth and silhouette images of wind turbine from its photos of multiple views using deep learning approaches, which aims for wind turbine blade fault diagnosis. Some previous multi-view based methods have extracted each photo’s silhouette and combined them into separate channels as the input of convolution; others use LSTM to combine a series of views for reconstruction. These approaches inevitably need a fixed number of views and the output result is divergent if the order of the input views is changed. So, we refer to a network, SiDeNet (Wiles and Zisserman, Learning to predict 3d surfaces of sculptures from single and multiple views. Int J Comp Vision, 2018), which has a flexible number of input views and will not be affected by the input order. It integrates both viewpoint and image information from each view to learn a latent 3D shape representation and use it to predict the depth of wind turbine at input views. Also, this representation could generalize to the silhouette of unseen views. We make the following contributions to SiDeNet: improving the resolution of predicted images by deepening network structure, adopting 6D camera pose to increase the degrees of freedom of viewpoint to capture a wider range of views, optimizing the loss function of silhouette by applying weights on edge points, and implementing silhouette refinement with point-wise optimizing. Additionally, we conduct a set of prediction experiments and prove the network’s generalization ability to unseen views. Evaluating predicted results on a realistic wind turbine dataset confirms the high performance of the network on both given views and unseen views.