The integrated mechanism of hierarchical decoders and dynamic fusion for image captioning

Abstract

Objective Image captioning aims at automatically generating lingual descriptions of images. It has a wide variety of applications scenarios like image indexing, medical imaging reports generation and human-machine interaction. To generate fluent sentences of the gathered information all, an algorithm of image captioning is called to recognize the scenes, entities and their relationships of the image. A deep encoder-decoder framework has been developed to resolve the issue past decades. The convolutional neural networks based (CNNs-based) encoder extracts feature vectors of the image and the recurrent neural networks based (RNNs-based) decoder generates image descriptions. Recent image captioning is driven by the development of attention mechanism. It improves the performance of image captioners via attending to informative image regions. Most attention models are based on the previously generated words as inputs when the next attending phases are predicted. Due to the lack of relevant textual guidance, most existing works are challenged of “attention defocus”, i. e., they fail to concentrate on correct image regions when generating the target words. As a result, contemporary models are prone to “hallucinating” objects, or missing informative visual clues, and make attention model be less interpretable. So, we facilitate an integrated hierarchical architecture and dynamic fusion strategy. Method The estimated word provides useful knowledge for predicting more grounded regions, although it is hard to localize the correct regions from the previously generated words at once. To refine the attention mechanism and improve the predicted words, we design a hierarchical architecture based on a series of captioning decoders. Our architecture is a hierarchical variant extended from the conventional encoder-decoder framework. Specifically, the first step is focused on the standard image captioning models, which generates a coarse description as a draft. To ground correct image regions with proper generated words, the latter one takes the outputs from the early decoder. Since the former decoder provides more predictable information to the target word, the attention accuracy is improved in latter decoders. To ground the final predicted words properly in this hierarchical architecture, attended regions from the early decoder can be well validated by the later decoders in a coarse-to-fine manner. Furthermore, we carry out a dynamic fusion strategy to aggregate the coarse-to-fine predictions from different decoders. Noteworthy, our manipulated gating mechanism is focused on the contributions from different decoders to the final word prediction. Differentiated from the previous gating mechanism managing the weight from each pathway, the contributions are jointed with a softmax schema from each decoder, which incorporates contextual information from all decoders to estimate the overall weight distribution. The dynamic fusion strategy provides rich fine-grained image descriptions and alleviates the problem of “vanishing gradients”, which makes the learning of the hierarchical architecture easier. Result Our method is evaluated on Microsoft common objects in context (MS COCO) and Flickr30K, which are the common benchmark for image captioning. The MS COCO dataset is composed of 120 k images, and the Flickr30K includes 31 k examples. Each image of both datasets is provided with five descriptions. The model is trained and tested using the Karpathy splits. The quantitative evaluation metrics are related to bilingual evaluation understudy (BLEU), metric for evaluation of translation with explicit ordering (MEREOR), and consensus-based image description evaluation (CIDEr). We compare the performance of our model with 12 current methods. On MS COCO, our analysis is optimized by 0. 5 and 1. 0 of each beyond BLEU-1 and CIDEr. Our result achieves a CIDEr of 69. 94 on Flickr30K. Compared to the baseline method (Transformer), our performance is optimized 4. 6 of CIDEr on MS COCO and 3. 8 on Flickr30K, which verifies that our method improves the accuracy of the predicted sentences effectively. In addition, our qualitative results demonstrate that the proposed method provides rich fine-grained image descriptions in comparison with other methods. Our method describes the number of appeared objects precisely when they belong to the same category. Our method could also describe small objects accurately. To further verify the effectiveness of the proposed hierarchical architecture, we visualize the attention mechanism and it shows that our method attends to discriminative parts of the target objects. In contrast, the baseline method may focus on irrelevant backgrounds, which leads to false predictions straightforward. Conclusion Our research is focused on a hierarchical architecture with dynamic fusion strategy for image captioning. The hierarchical architecture consists of a sequence of captioning decoders that refine the attention mechanism. To generate final sentence with rich fine-grained information, the dynamic fusion strategy aggregates different decoders. The ablation study demonstrates the effectiveness of each module in our proposed network. Our optimized results are demonstrated through the comparative experiments on MS COCO and Flickr30K datasets.