Multiple-level feature-based network for image captioning

Abstract

Image captioning, which automatically describes the content of an image, has attracted interests recently. Due to the need for both fine-grained visual understanding and meaningful natural language expression, image captioning is a challenging task. Existing methods predominantly take one kind of image feature to generate the description while neglecting other useful features. This strategy leads to unsatisfied captioning result. To deal with this problem, we propose a multiple-level feature-based network for image captioning. In our method, three kinds of features are extracted from the image, representing analysis of different level of the image. Attention mechanism in our network is adopted to selectively attend to salient region or attribute of each feature when predicting each word of the caption. Experimental results show that our model can outperform the state-of-the-art methods on MS-COCO dataset. Compared with other methods, our network can lead to more accurate subject prediction and vivid description of sentences.