A multi-task learning approach for image captioning

Abstract

In this paper, we propose a Multi-task Learning Approach for Image Captioning (MLAIC), motivated by the fact that humans have no difficulty performing such task because they have the capabilities of multiple domains. Specifically, MLAIC consists of three key components: (i) A multi-object classification model that learns rich category-aware image representations using a CNN image encoder; (ii) A syntax generation model that learns better syntax-aware LSTM based decoder; (iii) An image captioning model that generates image descriptions in text, sharing its CNN encoder and LSTM decoder with the object classification task and the syntax generation task, respectively. In particular, the image captioning model can benefit from the additional object categorization and syntax knowledge. The experimental results on MS-COCO dataset demonstrate that our model achieves impressive results compared to other strong competitors.