Pre-trained CNNs as Feature-Extraction Modules for Image Captioning: An Experimental Study

Abstract

Many recent image captioning works employ the Encoder-Decoder architecture, with Convolutional Neural Networks (CNNs) as feature extractors. This work presents a thorough experimental study about feature extraction using CNNs for the task of image captioning, in the context of deep learning. We examined 12 feature extraction architectures (from the VGG, ResNet, Inception, InceptionResNet, DenseNet, and NASNetLarge model families) and assessed their effectiveness as feature extractors using image captioning quality measures. The total is 72 experiments on 12 image classification CNNs, pre-trained on the ImageNet dataset. The features are extracted from the last layer after removing the fully connected layer and fed into the captioning model. We used a unified captioning model with a fixed vocabulary size across all the experiments to study the effect of changing the CNN feature extractor on image captioning quality. The scores are calculated using the standard metrics in image captioning. We found a strong relationship between the CNN model structure and the image captioning dataset, and that among the tested feature extraction CNNs, Xception and InceptionResNet V2 were the most robust while the two VGG models gave the least quality for image captioning. Based on these results, we recommend a set of pre-trained CNNs for each of the image captioning evaluation metrics that we want to optimise. To our knowledge, this work is the most comprehensive comparison between feature extractors for image captioning.